Idempotency Keys for Safe Retries

Every retry strategy discussed across SDK and client-side throttling has the same precondition: the request must be safe to send twice. For GET that is free; for POST it requires an idempotency key and a server that honours it. Without one, the most common rate-limiting outcome — a 429 on a payment, retried successfully — quietly becomes two charges.

The problem in concrete numbers

A client submits a payment. The server creates the charge, then the response is lost to a network drop at 30 seconds. The client sees a timeout, retries, and the server creates a second charge. Neither side is buggy: the client cannot distinguish “never arrived” from “arrived, response lost,” and the server has no way to recognise the second request as the same intent.

The same applies after a rejection. A 429 means the request was not processed — usually. But a limiter that rejects after the handler has already started, or a gateway that times out mid-write, leaves the same ambiguity. Retrying a write without a key is a bet that nothing happened.

A lost response with and without an idempotency key Without a key the retry creates a second charge because the server sees an unrelated request; with a key the server recognises the replay, skips the work, and returns the original response. The response never arrived — now what? no key POST /charges → charge created response lost in transit client retries the same body second charge created customer billed twice with Idempotency-Key POST + key → charge created response lost in transit client retries with the SAME key server replays the stored response one charge, correct result the key turns an ambiguous retry into a deterministic one

Key design reference

Decision Recommendation Why
Generation Client-side UUIDv4 or a hash of the intent The client is the only party that knows two attempts are one intent
Stability Generated once per logical operation, reused across all retries A new key per attempt defeats the mechanism entirely
Scope (account, endpoint, key) Prevents one tenant’s key colliding with another’s
Transport Idempotency-Key request header The de facto standard across payment and messaging APIs
Server storage Key → status, response body, status code, request fingerprint Enables both single-flight and replay
TTL 24 hours minimum Must outlive your longest retry schedule
In-flight handling Return 409 or block briefly Two concurrent attempts must not both execute
Body mismatch 422 with a clear error The same key with a different body is a client bug worth surfacing
Applies to POST, PATCH, and non-idempotent PUT GET, HEAD, and DELETE are already safe or naturally idempotent

The request fingerprint matters more than it looks. Storing a hash of the body lets the server distinguish a genuine replay from a client that reused a key for a different operation — a bug that would otherwise silently return the wrong response.

Step-by-step implementation

typescript
// client: the key belongs to the intent, not to the attempt.
export async function createCharge(input: ChargeInput) {
  const idempotencyKey = crypto.randomUUID();      // once, here — reused by every retry
  return withRetries(() =>
    apiFetch("/v1/charges", {
      method: "POST",
      headers: { "Content-Type": "application/json", "Idempotency-Key": idempotencyKey },
      body: JSON.stringify(input),
    }),
  );
}

// A retry helper that knows a 429 is safe to retry BECAUSE the key is stable.
async function withRetries(send: () => Promise<Response>, attempts = 5) {
  for (let n = 1; ; n += 1) {
    const res = await send();
    if (res.status !== 429 && res.status < 500) return res;
    if (n >= attempts) return res;
    const retryAfter = Number(res.headers.get("Retry-After") ?? 0);
    const backoff = retryAfter > 0
      ? retryAfter * 1000
      : Math.random() * Math.min(30_000, 500 * 2 ** (n - 1));   // full jitter
    await new Promise((r) => setTimeout(r, backoff));
  }
}
python
# server: single-flight, then replay. One transaction decides which path runs.
import hashlib, json
from datetime import timedelta
from django.db import IntegrityError, transaction
from django.http import JsonResponse

KEY_TTL = timedelta(hours=24)

def fingerprint(request) -> str:
    return hashlib.sha256(request.body).hexdigest()

def create_charge(request):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return JsonResponse({"error": "idempotency_key_required"}, status=400)

    scope = (request.account_id, "POST:/v1/charges", key)   # never global
    fp = fingerprint(request)

    with transaction.atomic():
        try:
            # Claim the key. The unique constraint IS the single-flight lock.
            record = IdempotencyRecord.objects.create(
                account_id=scope[0], endpoint=scope[1], key=scope[2],
                fingerprint=fp, status="in_progress",
            )
        except IntegrityError:
            record = IdempotencyRecord.objects.select_for_update().get(
                account_id=scope[0], endpoint=scope[1], key=scope[2])

            if record.fingerprint != fp:
                # Same key, different request: a client bug, not a replay.
                return JsonResponse(
                    {"error": "idempotency_key_reuse",
                     "message": "This Idempotency-Key was used with a different request body."},
                    status=422)

            if record.status == "in_progress":
                # A concurrent attempt is still running; tell the client to wait.
                return JsonResponse({"error": "request_in_progress"}, status=409,
                                    headers={"Retry-After": "2"})

            # Completed: replay the original outcome byte for byte.
            return JsonResponse(json.loads(record.response_body),
                                status=record.response_status,
                                headers={"Idempotent-Replay": "true"})

    # First time through: do the real work outside the claim transaction.
    charge = charge_service.create(request.account_id, json.loads(request.body))
    body, status = {"id": charge.id, "status": charge.status}, 201

    record.response_body = json.dumps(body)
    record.response_status = status
    record.status = "completed"
    record.expires_at = timezone.now() + KEY_TTL
    record.save(update_fields=["response_body", "response_status", "status", "expires_at"])
    return JsonResponse(body, status=status)
Four outcomes when a request carrying a key arrives A new key claims the record and does the work, a key already completed replays the stored response, a key still in progress returns a conflict with a retry hint, and a key reused with a different body is rejected as a client error. One lookup, four paths claim the key new do the work store the response 201 Created completed skip the work replay stored body same status as before in progress another attempt runs 409 + Retry-After never execute twice body differs key reused wrongly 422, explain it surface the client bug the unique constraint on the key is the lock — no separate locking service required Key scope and the collisions each choice allows A globally scoped key lets one tenant collide with another's, an account-scoped key still collides across endpoints, and a key scoped by account and endpoint together isolates every operation. Scope decides who can collide with whom global any tenant can claim another's key cross-tenant replay risk per account tenants isolated one key still spans unrelated endpoints account + endpoint every operation isolated fingerprint catches misuse the scope to ship the same UUID from two customers must never resolve to one stored response

Gotchas and edge cases

  • Generating the key in the transport layer. A key created inside the retry wrapper is new on every attempt, which is the same as having none. Create it where the intent is formed.
  • Losing the key across a process restart. A background job that regenerates its key after a crash duplicates work. Persist the key with the job record.
  • Global key scope. Keys scoped globally rather than per account allow one tenant to collide with another’s — accidentally or deliberately. Always scope by account and endpoint.
  • Storing only “seen”, not the response. Returning 409 for every replay forces clients to re-fetch state to discover what happened. Store the response and replay it.
  • TTL shorter than the retry schedule. A twelve-hour TTL with a twenty-four-hour retry window means late retries execute again. The key must outlive the retries.
  • Doing the work inside the claim transaction. Holding a database transaction open across an external payment call turns a slow provider into lock contention. Claim, commit, then work.
  • Ignoring keys on rejection paths. A 429 returned before the key is claimed is fine; one returned after the work started but before the record is written is the exact gap that produces duplicates. Claim first.

Verification and testing

bash
# 1. A replay must return the SAME resource, not a new one.
KEY=$(uuidgen)
first=$(curl -s -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
  -H "X-API-Key: $TEST_KEY" -d '{"amount":1200,"currency":"eur"}')
second=$(curl -s -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
  -H "X-API-Key: $TEST_KEY" -d '{"amount":1200,"currency":"eur"}')
[ "$(jq -r .id <<<"$first")" = "$(jq -r .id <<<"$second")" ] \
  && echo "replay OK: one charge" || echo "FAIL: duplicate charge created"

# 2. Same key, different body must be refused rather than replayed.
curl -s -o /dev/null -w "%{http_code}\n" -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $KEY" -H "Content-Type: application/json" \
  -H "X-API-Key: $TEST_KEY" -d '{"amount":9900,"currency":"eur"}'
# expect 422

# 3. Concurrent attempts with one key must produce exactly one charge.
KEY2=$(uuidgen)
seq 1 8 | xargs -P8 -I{} curl -s -o /dev/null -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $KEY2" -H "Content-Type: application/json" \
  -H "X-API-Key: $TEST_KEY" -d '{"amount":500,"currency":"eur"}'
# then confirm exactly one charge exists for that key

Test three is the one that finds broken implementations. Eight concurrent attempts with one key must produce one charge, some number of 409s, and no duplicates — anything else means the claim is not atomic.

Frequently Asked Questions

Where should the idempotency key be generated?

At the point the intent is created — the button handler, the queued job, the workflow step — and then reused by every attempt. Generating it inside the retry helper produces a new key per attempt, which removes the protection entirely while appearing to implement it.

Do I need keys if the API only ever returns 429 before processing?

You need them for the cases you cannot see: a lost response, a proxy timeout, a limiter that rejects after the handler began. Any of those makes a retry ambiguous. If an operation charges money, creates a resource, or sends a message, treat retries as unsafe without a key.

How long should keys be retained?

Longer than the longest retry schedule any client could follow — twenty-four hours is a common floor, and payment APIs often keep them longer. Expiring earlier means a late retry executes as a fresh request, which is precisely the duplicate you were preventing.

What should the server do while a first attempt is still running?

Return a conflict with a short Retry-After rather than executing concurrently or blocking for a long time. The client retries in a couple of seconds and gets the replayed response. Blocking the second request until the first finishes also works, but ties up a connection and behaves badly when the first attempt is slow.