System design · Payments · Part 2
A simple payment architecture
A first architecture, its API and happy path, what happens when a step fails, and how idempotency keys prevent double charges.
A first, simple architecture
Start with the simplest system that works. The customer clicks Pay, the frontend calls the payment service over HTTPS, and the payment service saves the payment in its database and asks Stripe to process it.
Customer
Clicks Pay
- HTTPS
Frontend
Payment service
- save the payment
Database
- process the payment
Stripe
The basic version: no Redis, no Kafka, no microservices.
Designing the API
Suppose an order already exists: order_123, with a total of $100. To pay for it, the frontend creates a payment:
{
"orderId": "order_123"
}1.Why the amount isn't in the request
Never trust the client. Anything the browser sends can be edited before it leaves the customer's computer. If the request carried the amount, a malicious user could change it:
Amount from the browser
POST /v1/payments
{
"orderId": "order_123"
"amount": 1000
}
- The real total is $100 (10000), but the request was edited to $10 (1000).
- A service that trusts it charges $10 for a $100 order.
Amount from the order service
POST /v1/payments
{
"orderId": "order_123"
}
- The request only says which order to pay.
- The payment service looks up the real amount itself.
So the payment service gets the trusted amount from the order service, or other trusted backend data:
Frontend
order_123
Payment service
- get the order
Order service
- amount: $100
Payment service
- charge $100
Stripe
The architecture
Putting it together, with an API gateway in front. In many architectures, the API gateway is the single entry point into the backend services.
Customer
Frontend
- POST /v1/payments
API gateway
Payment service
- order details
Order service
- payment records
Payment DB
- payment processing
Stripe
The payment service reads the order, keeps its own records, and charges through Stripe.
The happy path
One complete payment where nothing goes wrong. A customer buys a MacBook for $1,500.
1.The order is created
The order service creates the order:
order_id: order_555
amount: 150000
currency: USD
status: PENDING_PAYMENT2.The customer clicks Pay
The frontend sends only the order ID:
{
"orderId": "order_555"
}3.Verify the order
The payment service asks the order service for order_555, and gets back the trusted amount:
{
"id": "order_555",
"amount": 150000,
"currency": "USD",
"status": "PENDING_PAYMENT"
}4.Create the payment
The payment service saves a new payment:
payment_id: pay_999
order_id: order_555
amount: 150000
currency: USD
status: CREATED5.Charge through Stripe
The status moves to PROCESSING, and the payment service asks Stripe to process the payment. Stripe returns success.
6.Mark it paid
The payment service updates the status to SUCCEEDED, and the order service marks the order PAID.
Customer
Frontend
Payment service
- verify order
Order service
Payment service
- create payment
Payment DB
Payment service
- charge
Stripe
- success
Payment service
- mark succeeded
Payment DB
Order service
PAID
The whole happy path, top to bottom.
The happy path is straightforward. System design gets interesting when something fails.
When things fail
1.A crash after the charge
Consider this sequence:
- The customer clicks Pay.
- The payment service sends the request to Stripe.
- Stripe charges the card successfully.
- Stripe sends back a success response.
- The payment service crashes before saving
SUCCEEDED.
Now the two sides disagree:
Reality
The outside world says the customer paid.
The payment database
The database still says the result is unknown.
This is an inconsistent state, and it's a fundamental distributed systems problem. Charging through Stripe and updating the database are two separate operations, and there's no way to make them one.
2.Why a database transaction doesn't help
If everything lived inside PostgreSQL, a transaction would make both updates happen together, or neither:
BEGIN;
UPDATE payments ...;
UPDATE orders ...;
COMMIT;But Stripe is an external service, outside the database. There is no ordinary transaction like this:
BEGIN;
Charge the card through Stripe and Visa
UPDATE payments ...;
COMMIT;That's why distributed systems are hard:
- Network calls can fail
- Servers can crash
- Responses can be lost
- Requests can be repeated
3.Duplicate clicks
The customer clicks Pay, and nothing happens for two seconds. Thinking it didn't work, they click again. The frontend sends two requests for the same $100 order:
Without protection
Charge $100
Charge $100
Each request becomes its own charge: $200 for a $100 order.
What should happen
Charge $100
However many requests arrive, the customer is charged once.
Preventing this leads to one of the most important concepts in payment system design: idempotency.
Idempotency
An operation is idempotent when requesting it several times has the same effect as requesting it once.
If Pay order_123 arrives five times, the result should be one $100 charge, not five.
- Pay order_123
- Pay order_123
- Pay order_123
- Pay order_123
- Pay order_123
One $100 charge
Not five
1.Idempotency keys
A common solution is an idempotency key. The frontend creates a unique identifier for the checkout attempt, like checkout_order_123_attempt_1, or more commonly a UUID, and sends it with the request:
POST /v1/payments
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000The payment service saves the key with the payment:
payment_id: pay_123
idempotency_key: 550e8400...
status: SUCCEEDEDIf a request arrives with the same key, the payment service finds it in the database: this was already processed. Instead of charging again, it returns the previous result.
2.The flow
First request
Request
key = abc123
Payment service
Database
Key exists? No
Create payment
Charge Stripe
Same request again
Request
key = abc123
Payment service
Database
Key exists? Yes
Return payment
No new charge
The second request never reaches Stripe.
The race condition
Checking first and then creating leaves a gap. Imagine two identical requests, A and B, arriving at almost exactly the same time:
| Request A | Request B | |
|---|---|---|
| 1 | Does abc123 exist? | Does abc123 exist? |
| 2 | No | No |
| 3 | Create payment | Create payment |
| 4 | Charge Stripe | Charge Stripe |
Both checks ran before either payment was saved, so both said no. The customer is charged twice again. This is a race condition.
1.The fix: a unique constraint
Let the database enforce it. A unique constraint on the key means only one row with abc123 can ever exist:
UNIQUE (idempotency_key)Now the insert itself is the check, and the database lets only one request win:
| Request A | Request B | |
|---|---|---|
| 1 | INSERT abc123: succeeds | INSERT abc123: duplicate key |
| 2 | Charge Stripe | Fetch the existing payment and return it |
Idempotency with Stripe
There is one more layer. Even with an idempotent backend, this can happen:
- The payment service sends the charge to Stripe.
- Stripe processes it successfully.
- The network breaks before the response reaches the payment service.
- The payment service can't tell whether Stripe got the request, so it retries.
Without idempotency at the provider, that retry is a second charge. So when the provider supports it, the payment service sends a stable idempotency key to the provider too, such as its own payment ID:
Idempotency-Key: pay_123The first call processes $100. A retry with the same key, pay_123, is recognized as the same operation, and Stripe doesn't create another charge.
Client
- idempotency keyidempotency key
Payment service
- idempotency keyidempotency key
Stripe
Protection at two levels.