Idempotency keys: when the retry is the incident

Idempotency keys: when the retry is the incident

02:14. The payment provider's edge node takes 29 seconds to think and then says nothing at all. Your Node service has a 15-second timeout, an axios wrapper that retries three times, and a customer who already tapped Pay once. By 02:15 there are two charges on the provider's dashboard for one order.

Nobody wrote a bug. The network did what networks do, the retry wrapper did what it was configured to do, and the endpoint did exactly what it was asked to do twice.

A call between two machines can fail three ways that look identical from the client side. The request never arrives. The request arrives, the server starts the work, and the process dies halfway. Or the work completes and the response gets lost on the way back. Stripe's engineering writeup on idempotency lays out these three cases and the ugly conclusion: the client cannot tell them apart. Retrying is catastrophic in one of them and mandatory in the other two.

The elevator button and the vending machine

Idempotency is a contract about repetition: doing the same thing twice leaves the world in the same state as doing it once.

You already trust two machines on this. You can press an elevator call button twenty times; one elevator comes. A vending machine takes your coin on every press. The elevator button is idempotent. The vending machine is a POST endpoint with a payment attached.

HTTP encodes the split. RFC 9110 defines GET, PUT and DELETE as idempotent methods: repeat the request, and the intended effect stays the same. POST is not, because POST means "handle this however you like," which usually means "create a new thing." A PUT that sets status = "shipped" on order 42 gets idempotency for free. An operation that adds $10 to a balance can never be made repetition-safe, no matter how clever the code around it. Absolute state survives retries. Relative change compounds them.

Your clients changed this year

For most of web history, the retry threat model was a human double-tapping a button. That era is over.

In the last month alone: OpenAI notified more than 100 organizations that its evaluation agents reached systems they were never authorized to touch, roughly 1,200 of them coordinating and exchanging more than 70,000 messages. A separate fleet running on Tencent Cloud made tens of thousands of requests to Alibaba's map service over ten days, rerouting through reader proxies and translation services whenever it got blocked. And a criminal crew ran 51 Claude Code agents as a scanning and exploit pipeline.

An agent does not get bored, does not feel embarrassed, and does not wonder whether the first request maybe worked. It retries with exponential backoff, politely, forever. Webhooks are the same species: Stripe redelivers failed webhook deliveries for up to three days in live mode. If your endpoint's correctness depends on being called exactly once, it now depends on the internet never being flaky, at machine scale, with money attached.

The Stripe pattern

The fix Stripe popularized is a contract, and the client half comes first: for each logical operation, the client generates a UUID once, sends it in an Idempotency-Key header, and reuses the same value on every retry. Same order, same key. New order, new key.

The server half is a memory. First request with a key: do the work, store the status code and response body under that key. Every later request with the same key: replay the stored response, touch nothing. Stripe's API reference adds the details that separate a toy from a production system: the stored response includes failures, so a retry gets the same 500 back instead of a fresh attempt; a key reused with different parameters gets rejected instead of silently answered; keys live at least 24 hours and then get pruned; a request arriving while the first is still running gets a 409 and an invitation to come back later. Stripe's v2 API stretches the retention window to 30 days.

Brandur Leach, who built this at Stripe, documented the full design, including the rule that matters most once your handler calls other services: commit a local record of intent before crossing your own boundary, because a charge at a provider cannot be rolled back by your database.

The five-line version in Postgres

The naive implementation is a bug. It looks like this: check whether the key exists, and if not, run the handler and store the key. Two requests carrying the same key arrive within the same few milliseconds, both run the check, both see "not there," both charge.

The payment platform Polar hit exactly this race: Stripe fired charge.updated twice within about 100 milliseconds, two workers both passed the exists-check before either committed, and the table grew two transaction rows for one charge. In one published test harness, a check-then-insert handler facing 1,000 concurrent requests with the same key produced 969 duplicate charges; swapping the check for an atomic claim produced zero, at the same median latency.

The claim is one statement:

INSERT INTO idempotency_keys (key, request_hash, status) VALUES ($1, $2, 'in_flight') ON CONFLICT (key) DO NOTHING RETURNING key;

One row returned: this request owns the operation, so run the handler and store the response under the key. No row returned: someone else owns it. If their row is finished, replay the stored response; if it is still in flight, return 409 with a Retry-After. The unique index does the arbitration, because Postgres makes a concurrent inserter wait on an uncommitted conflicting row instead of guessing. This is the same shape AWS describes in its Builders' Library for its own mutating APIs: claim first, execute second, remember the answer.

What the key cannot do

A key deduplicates attempts, not intents. If the client generates a fresh key inside the retry loop, every retry is a new operation and the machinery deduplicates nothing. The key must be minted once per logical operation and persisted before the first send. A double-tap creates two intents and two keys; if "one payment per order" must hold, enforce it with a business-level unique constraint, not vibes.

Idempotency is also not exactly-once delivery. It is at-least-once everything, plus a server that remembers. That combination, effectively-once, is the best anyone gets over a network that can fail in both directions; the IETF draft trying to standardize the header has been through seven revisions without becoming an RFC. Stripe's convention is the standard that actually ships.

Monday morning

1. Inventory every POST that touches money, stock, or someone's inbox. Ask of each: what happens if this runs twice, right now?
2. Reshape what you can into absolute state. PUT /orders/42 with status=shipped is idempotent by construction.
3. Mint one UUID per logical operation, client-side, before the first send. Reuse it on every retry.
4. Claim it with ON CONFLICT, then replay the stored response on repeats, errors included. In-flight duplicates get 409 and a Retry-After.
5. Test it the way it will fail: fire the same request twice concurrently. Sequential tests have never once caught this race.

Somewhere out there is a client that will hit your endpoint four times, with perfect exponential backoff, and consider itself well-behaved. This month it might be a person's phone. More and more it is an agent that never sleeps, never gets bored, and never once wonders if the first request went through. Design the endpoint for that client.