System design · Payments · Part 3
Webhooks
How a payment service learns the result when a response is lost: webhooks, eventual consistency, duplicate events and signature checks.
Lost responses
The payment service asks Stripe to charge $100. Stripe processes it successfully, but the response disappears on the way back:
Payment service
- charge $100charge $100
Stripe
Succeeded
- ✕ response lost✕ response lost
Payment service
Still PROCESSING
Stripe has charged the card, but the payment service never hears about it.
The payment stays PROCESSING in the database. How does the payment service eventually learn that the payment succeeded? One powerful answer is a webhook.
What is a webhook
A webhook is another server calling this server when something happens. Normally, the payment service calls Stripe. A webhook reverses the direction:
A normal request
Payment service
Stripe
A webhook
Stripe
Payment service
To say a payment succeeded, Stripe sends a request to an endpoint like POST /webhooks/stripe, with an event that conceptually looks like this:
{
"type": "payment.succeeded",
"paymentId": "pi_abc123"
}Even when the original response is lost, Stripe can report the result separately.
Why webhooks matter
Replay the failure from before:
- The payment service calls Stripe.
- Stripe charges the customer.
- The response gets lost.
- The payment stays
PROCESSING.
A few seconds later, the webhook arrives:
Stripewebhook
Payment service
- pi_abc123 succeeded
Update database
SUCCEEDED
The webhook carries the result the lost response would have.
The system becomes correct, eventually. This is called eventual consistency.
Eventual consistency
Systems may temporarily disagree, but eventually they converge to the correct state.
Before the webhook
For a moment, the two sides disagree.
After the webhook
The webhook brings them back in line.
Payment systems use this idea often, because external services and networks are not perfectly reliable.
Webhooks arrive more than once
Payment providers usually retry webhooks. Imagine Stripe sends payment.succeeded, but the payment service is temporarily down. Stripe doesn't know whether the event was handled, so it sends it again, and maybe again:
- Webhook 1: evt_123
- Webhook 2: evt_123
- Webhook 3: evt_123
- Webhook 4: evt_123
Processed once
The same event, four deliveries
So webhook processing must be idempotent too. The same concept keeps appearing.
1.Webhook idempotency
Each event from the provider usually has an ID, like evt_123. The payment service records every event it has handled:
| webhook_events | Meaning |
|---|---|
| event_id | The provider's ID for the event |
| provider | Stripe |
| processed_at | When it was handled |
When an event arrives, the first question is: has evt_123 already been processed?
New event
Webhook
evt_123
Database
Processed? No
Process it
Same event again
Webhook
evt_123
Database
Processed? Yes
Ignore duplicate
A repeated event changes nothing.
Again, a database unique constraint closes the race between two deliveries arriving at once:
UNIQUE (provider, event_id)Is the webhook really from Stripe?
The webhook endpoint is a public URL. Without a check, an attacker could send this and make an unpaid order appear paid:
POST /webhooks/stripe
{ "type": "payment.succeeded" }So providers typically sign webhook requests. Stripe sends the event body with a signature, and the payment service verifies the signature using a secret shared with Stripe. Only after verification is the event trusted.
Stripe
Event body + signature
Webhook endpoint
Verifies with the secret
Always
Receive webhook
Verify signature
Check duplicate event
Process event
Update payment
Never
Receive webhook
Blindly trust it
Every webhook goes through the same steps, in this order.
The updated architecture
The architecture is becoming realistic:
Customer
Frontend
- HTTPS
Payment service
- save
PostgreSQL
- process
Stripe
- webhook
Webhook endpoint
PostgreSQL
Stripe now reports results back through the webhook endpoint.