System design · Payments · Part 4

Asynchronous work and the outbox pattern

What has to happen before responding and what can wait, why a message broker helps, and how the transactional outbox keeps events from getting lost.

Synchronous and asynchronous

When a customer clicks Pay, some work needs to happen immediately:

  • Validate the request
  • Check the order
  • Create the payment
  • Start the provider request

This is synchronous: the client waits for the response.

Other work doesn't need to happen before responding:

  • Send the receipt email
  • Update analytics
  • Write audit events
  • Notify the warehouse
  • Add loyalty points
  • Fraud post-processing

Making the customer wait for all of that would be wasteful. So it runs asynchronously, in the background:

  1. Payment succeeded

  • now

    Respond to customer

  • later

    Queue background work

The customer gets an answer right away; the rest happens after.

Why use a message queue

After a payment succeeds, several things should happen:

ServiceAction
Order serviceMark the order paid
Email serviceSend the receipt
AnalyticsRecord the purchase
InventoryUpdate the stock
RewardsAdd points

1.Calling each service

One simple implementation is for the payment service to call each service itself:

  1. Payment service

  • Order service

  • Email service

    Down

  • Analytics service

    Slow

  • Inventory service

  • Rewards service

One of the five is down. What should the payment service do?

The problems show up when a service misbehaves:

  • A service is down. Should the payment fail? No, the payment already succeeded.
  • Analytics is slow. Should the customer wait? No.

2.A message broker

This motivates asynchronous messaging. The payment service publishes one event to a message broker, and each service consumes it independently:

  1. Payment service

  2. PaymentSucceeded event

    Message broker

  • Order service

  • Email service

  • Analytics service

  • Rewards service

The payment service doesn't need to know who's listening, or whether they're up.

Common message brokers include:

  • Kafka

  • RabbitMQ

  • Amazon SQS

  • Google Pub/Sub

Don't add Kafka by default

Naming a technology isn't a design. Explain the decision, and the problem it solves:

Weak

“I'll use Kafka.”

Names a tool, but not why.

Strong

“Once a payment succeeds, it triggers several independent downstream actions. I'd publish a PaymentSucceeded event through a durable message broker. That decouples the payment request from slower or temporarily unavailable consumers.”

Solves a problem instead of naming a technology.

A hidden problem

The payment service now does two things: it updates the payment to SUCCEEDED in PostgreSQL, and it publishes PaymentSucceeded to Kafka. What if the database update succeeds, but the publish fails?

Payment DB

pay_123SUCCEEDED

The payment succeeded.

Order service

PaymentSucceededNever arrives

The order is never marked paid.

Distributed systems again: a database write and a message publish are two separate operations, but they need to happen together. This leads to the transactional outbox pattern.

The transactional outbox pattern

Instead of updating the database and then publishing to Kafka, write both the payment update and an event record in the same database transaction. The payments table gets:

payments
pay_123 → SUCCEEDED

And at the same time, outbox_events gets:

outbox_events
id:         event_456
type:       PaymentSucceeded
payment_id: pay_123
published:  false

Both happen in one transaction:

SQL
BEGIN;

UPDATE payments
SET status = 'SUCCEEDED'
WHERE id = 'pay_123';

INSERT INTO outbox_events (...);

COMMIT;

Now either both succeed, or both fail. There's no state where the payment succeeded but the event was forgotten.

1.The outbox worker

A background worker reads the rows in outbox_events where published = false, and publishes them to Kafka or SQS.

  1. Payment service

  2. one DB transaction

    PostgreSQL

    payment = SUCCEEDED outbox = PaymentSucceeded

  3. background worker

    Message broker

  • Order service

  • Email service

  • Analytics service

The event is saved with the payment, then published by the worker.

If publishing fails, published stays false, and the worker retries later. Much safer.

What does exactly once mean

The goal is one payment, handled exactly once. But distributed systems make true exactly-once processing difficult: Kafka, SQS and webhooks may all deliver an event more than once.

That's fine, as long as consumers handle duplicates safely. PaymentSucceeded might arrive three times:

  • PaymentSucceeded
  • PaymentSucceeded
  • PaymentSucceeded

  • Order service

    Already PAID. Do nothing.

  • Email service

    Event seen. One receipt.

Each consumer makes a repeated event harmless.

  • The order service sees the order is already PAID, and does nothing.
  • The email service can use the event ID, so it sends only one receipt.