System design · Payments · Part 2

A simple payment architecture

A first architecture, its API and happy path, what happens when a step fails, and how idempotency keys prevent double charges.

A first, simple architecture

Start with the simplest system that works. The customer clicks Pay, the frontend calls the payment service over HTTPS, and the payment service saves the payment in its database and asks Stripe to process it.

  1. Customer

    Clicks Pay

  2. HTTPS

    Frontend

  3. Payment service

  • save the payment

    Database

  • process the payment

    Stripe

The basic version: no Redis, no Kafka, no microservices.

Designing the API

Suppose an order already exists: order_123, with a total of $100. To pay for it, the frontend creates a payment:

POST /v1/payments
{
  "orderId": "order_123"
}

1.Why the amount isn't in the request

Never trust the client. Anything the browser sends can be edited before it leaves the customer's computer. If the request carried the amount, a malicious user could change it:

Amount from the browser

POST /v1/payments

{

"orderId": "order_123"

"amount": 1000

}

  • The real total is $100 (10000), but the request was edited to $10 (1000).
  • A service that trusts it charges $10 for a $100 order.

Amount from the order service

POST /v1/payments

{

"orderId": "order_123"

}

  • The request only says which order to pay.
  • The payment service looks up the real amount itself.

So the payment service gets the trusted amount from the order service, or other trusted backend data:

  1. Frontend

    order_123

  2. Payment service

  3. get the order

    Order service

  4. amount: $100

    Payment service

  5. charge $100

    Stripe

The architecture

Putting it together, with an API gateway in front. In many architectures, the API gateway is the single entry point into the backend services.

  1. Customer

  2. Frontend

  3. POST /v1/payments

    API gateway

  4. Payment service

  • order details

    Order service

  • payment records

    Payment DB

  • payment processing

    Stripe

The payment service reads the order, keeps its own records, and charges through Stripe.

The happy path

One complete payment where nothing goes wrong. A customer buys a MacBook for $1,500.

1.The order is created

The order service creates the order:

order_id: order_555
amount:   150000
currency: USD
status:   PENDING_PAYMENT

2.The customer clicks Pay

The frontend sends only the order ID:

POST /v1/payments
{
  "orderId": "order_555"
}

3.Verify the order

The payment service asks the order service for order_555, and gets back the trusted amount:

Order service response
{
  "id": "order_555",
  "amount": 150000,
  "currency": "USD",
  "status": "PENDING_PAYMENT"
}

4.Create the payment

The payment service saves a new payment:

payment_id: pay_999
order_id:   order_555
amount:     150000
currency:   USD
status:     CREATED

5.Charge through Stripe

The status moves to PROCESSING, and the payment service asks Stripe to process the payment. Stripe returns success.

6.Mark it paid

The payment service updates the status to SUCCEEDED, and the order service marks the order PAID.

  1. Customer

  2. Frontend

  3. Payment service

  4. verify order

    Order service

  5. Payment service

  6. create payment

    Payment DB

  7. Payment service

  8. charge

    Stripe

  9. success

    Payment service

  10. mark succeeded

    Payment DB

  11. Order service

    PAID

The whole happy path, top to bottom.

The happy path is straightforward. System design gets interesting when something fails.

When things fail

1.A crash after the charge

Consider this sequence:

  1. The customer clicks Pay.
  2. The payment service sends the request to Stripe.
  3. Stripe charges the card successfully.
  4. Stripe sends back a success response.
  5. The payment service crashes before saving SUCCEEDED.

Now the two sides disagree:

Reality

StripeSucceeded
Customer's bankCharged

The outside world says the customer paid.

The payment database

payments.statusPROCESSING

The database still says the result is unknown.

This is an inconsistent state, and it's a fundamental distributed systems problem. Charging through Stripe and updating the database are two separate operations, and there's no way to make them one.

2.Why a database transaction doesn't help

If everything lived inside PostgreSQL, a transaction would make both updates happen together, or neither:

SQL
BEGIN;
UPDATE payments ...;
UPDATE orders ...;
COMMIT;

But Stripe is an external service, outside the database. There is no ordinary transaction like this:

Not possible
BEGIN;
Charge the card through Stripe and Visa
UPDATE payments ...;
COMMIT;

That's why distributed systems are hard:

  • Network calls can fail
  • Servers can crash
  • Responses can be lost
  • Requests can be repeated

3.Duplicate clicks

The customer clicks Pay, and nothing happens for two seconds. Thinking it didn't work, they click again. The frontend sends two requests for the same $100 order:

Without protection

Charge $100

Charge $100

Charged$200

Each request becomes its own charge: $200 for a $100 order.

What should happen

Charge $100

Charged$100

However many requests arrive, the customer is charged once.

Preventing this leads to one of the most important concepts in payment system design: idempotency.

Idempotency

An operation is idempotent when requesting it several times has the same effect as requesting it once.

If Pay order_123 arrives five times, the result should be one $100 charge, not five.

  • Pay order_123
  • Pay order_123
  • Pay order_123
  • Pay order_123
  • Pay order_123

One $100 charge

Not five

1.Idempotency keys

A common solution is an idempotency key. The frontend creates a unique identifier for the checkout attempt, like checkout_order_123_attempt_1, or more commonly a UUID, and sends it with the request:

Request
POST /v1/payments
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000

The payment service saves the key with the payment:

Payment DB
payment_id:      pay_123
idempotency_key: 550e8400...
status:          SUCCEEDED

If a request arrives with the same key, the payment service finds it in the database: this was already processed. Instead of charging again, it returns the previous result.

2.The flow

First request

  1. Request

    key = abc123

  2. Payment service

  3. Database

    Key exists? No

  4. Create payment

  5. Charge Stripe

Same request again

  1. Request

    key = abc123

  2. Payment service

  3. Database

    Key exists? Yes

  4. Return payment

    No new charge

The second request never reaches Stripe.

The race condition

Checking first and then creating leaves a gap. Imagine two identical requests, A and B, arriving at almost exactly the same time:

Request ARequest B
1Does abc123 exist?Does abc123 exist?
2NoNo
3Create paymentCreate payment
4Charge StripeCharge Stripe

Both checks ran before either payment was saved, so both said no. The customer is charged twice again. This is a race condition.

1.The fix: a unique constraint

Let the database enforce it. A unique constraint on the key means only one row with abc123 can ever exist:

SQL
UNIQUE (idempotency_key)

Now the insert itself is the check, and the database lets only one request win:

Request ARequest B
1INSERT abc123: succeedsINSERT abc123: duplicate key
2Charge StripeFetch the existing payment and return it

Idempotency with Stripe

There is one more layer. Even with an idempotent backend, this can happen:

  1. The payment service sends the charge to Stripe.
  2. Stripe processes it successfully.
  3. The network breaks before the response reaches the payment service.
  4. The payment service can't tell whether Stripe got the request, so it retries.

Without idempotency at the provider, that retry is a second charge. So when the provider supports it, the payment service sends a stable idempotency key to the provider too, such as its own payment ID:

Request to Stripe
Idempotency-Key: pay_123

The first call processes $100. A retry with the same key, pay_123, is recognized as the same operation, and Stripe doesn't create another charge.

  1. Client

  2. idempotency key

    Payment service

  3. idempotency key

    Stripe

Protection at two levels.