Idempotency and Retries skill for Claude Code

A client that times out does not know whether the operation happened. Done well, it can retry with the same key and the server guarantees the effect happens once and returns the same result; and retries are bounded, jittered, and happen at one layer so they do not turn a blip into an outage.

Ground rules

• Every non-idempotent operation exposed to retries (POST that creates or charges, message handlers with side effects) needs an idempotency key or natural idempotence.
• The client generates the key once per logical operation, before the first attempt, and reuses it for every retry.
• Retry only errors that are transient and only operations that are idempotent (or keyed).
• Every remote call has a timeout. Every retry loop has a max attempt count and an overall deadline.
• Retry at one layer of the stack. Retries at three layers of three attempts each make 27 calls to a struggling dependency.

Workflow

- [ ] 1. List operations and classify: naturally idempotent, keyed, or unsafe
- [ ] 2. Add natural idempotence where cheap (upsert, conditional update, PUT)
- [ ] 3. Add idempotency keys to the rest; store results atomically
- [ ] 4. Propagate keys to downstream side effects
- [ ] 5. Define the retry policy per client: retryable errors, backoff, deadline, budget
- [ ] 6. Remove duplicate retry layers
- [ ] 7. Test duplicates, concurrency, timeouts, and crashes mid-operation

2. Natural idempotence (prefer when possible)

Technique	Example
Upsert by business key	INSERT ... ON CONFLICT (order_id, line_no) DO UPDATE ...
Guarded state transition	UPDATE orders SET status='paid' WHERE id=$1 AND status='pending' (0 rows = already done)
Full replacement	PUT /users/42/settings with the whole document
Conditional requests	If-Match: "<etag>" → 412 if changed; prevents lost updates on retry
Version check for events	Apply only if event.version > row.version
Set semantics	SADD, DELETE ... WHERE id=$1 (deleting twice is fine)

3. Idempotency keys

Client sends Idempotency-Key: <uuid> (the header name used by Stripe and an IETF draft). Server behavior:

Situation	Response
New key	Process; store request fingerprint and final response
Same key, same payload, completed	Replay the stored status and body; no side effects
Same key, same payload, still in progress	409 Conflict (client retries later)
Same key, different payload	422 Unprocessable Content (key misuse)
Key required but missing	400 Bad Request

Storage (Postgres):

CREATE TABLE idempotency_keys (
  scope         text NOT NULL,             -- tenant or user id; keys are not global
  key           text NOT NULL,
  request_hash  text NOT NULL,             -- hash of method + path + canonical body
  state         text NOT NULL CHECK (state IN ('in_progress', 'completed')),
  response_code int,
  response_body jsonb,
  locked_until  timestamptz NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (scope, key)
);

Handler flow:

async function withIdempotency(scope: string, key: string, reqHash: string, handler: (tx: Tx) => Promise<Resp>) {
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (scope, key, request_hash, state, locked_until)
     VALUES ($1, $2, $3, 'in_progress', now() + interval '60 seconds')
     ON CONFLICT DO NOTHING RETURNING 1`, [scope, key, reqHash]);

  if (claimed.rowCount === 0) {
    const row = await db.one('SELECT * FROM idempotency_keys WHERE scope=$1 AND key=$2', [scope, key]);
    if (row.request_hash !== reqHash) return { status: 422, body: { error: 'idempotency key reused with different request' } };
    if (row.state === 'completed') return { status: row.response_code, body: row.response_body };
    if (row.locked_until > new Date()) return { status: 409, body: { error: 'request in progress' } };
    // lock expired: previous attempt crashed; fall through and re-run (handler must be safe to resume)
  }

  try {
    return await db.transaction(async (tx) => {
      const resp = await handler(tx);                         // business writes in the same transaction
      await tx.query(
        `UPDATE idempotency_keys SET state='completed', response_code=$3, response_body=$4
         WHERE scope=$1 AND key=$2`, [scope, key, resp.status, resp.body]);
      return resp;
    });
  } catch (err) {
    // transaction rolled back: nothing happened, so release the key for a clean retry
    await db.query('DELETE FROM idempotency_keys WHERE scope=$1 AND key=$2 AND state=$3', [scope, key, 'in_progress']);
    throw err;
  }
}

• Store final business outcomes (2xx and deterministic 4xx). Do not store results for requests rejected before execution or for unexpected errors whose transaction rolled back; release the key instead.
• Expire keys after a documented window (24 h is common; Stripe uses 24 h).
• Scope keys per tenant/user so one client cannot replay another’s response.
• Multi-step operations with external calls (charge card, then create shipment) need recovery points: record progress per step so a retry resumes rather than repeats.

4. Downstream side effects

• Pass a derived key to every provider that supports one: Idempotency-Key: ${key}:charge to the payment provider.
• For providers without keys, record the external call’s result before acting on it, and look up by your own reference (e.g., provider metadata.order_id) before retrying.
• Emails and notifications: dedupe on (template, entity_id, key) in an outbox.
• Message consumers: see the inbox pattern in event-driven-architecture.

5. Retry policy

Retryable vs. not:

Retry	Do not retry
Connection refused/reset, DNS failure, timeouts (for idempotent/keyed ops)	400, 401, 403, 404, 409 (unless documented), 422
429 (honor Retry-After)	Validation or business-rule errors
502, 503, 504	500 from a non-idempotent call without a key
gRPC UNAVAILABLE, RESOURCE_EXHAUSTED (with backoff)	gRPC INVALID_ARGUMENT, PERMISSION_DENIED, FAILED_PRECONDITION

Backoff with full jitter: sleep = random(0, min(cap, base * 2^attempt)).

export async function retry<T>(fn: (attempt: number) => Promise<T>, opts = { attempts: 4, baseMs: 100, capMs: 5_000, deadlineMs: 15_000 }) {
  const start = Date.now();
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn(attempt);
    } catch (err: any) {
      const retryAfterMs = parseRetryAfter(err?.response?.headers?.['retry-after']);
      const backoff = retryAfterMs ?? Math.random() * Math.min(opts.capMs, opts.baseMs * 2 ** attempt);
      const outOfTime = Date.now() - start + backoff > opts.deadlineMs;
      if (!isRetryable(err) || attempt + 1 >= opts.attempts || outOfTime) throw err;
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
}

// usage: key created once, outside the loop
const key = crypto.randomUUID();
await retry(() => api.post('/payments', body, { headers: { 'Idempotency-Key': key }, timeout: 3_000 }));

Python with tenacity:

from tenacity import retry, stop_after_attempt, stop_after_delay, wait_random_exponential, retry_if_exception

@retry(stop=(stop_after_attempt(4) | stop_after_delay(15)),
       wait=wait_random_exponential(multiplier=0.1, max=5),
       retry=retry_if_exception(is_transient), reraise=True)
def create_payment(body, key):
    return client.post("/payments", json=body, headers={"Idempotency-Key": key}, timeout=3)

Policy defaults to start from: 3–5 attempts, base 100–200 ms, cap 5–10 s, overall deadline below the caller’s own timeout.

6. Avoid amplification

• Retry at the outermost layer that can make the decision (often the edge client or the job runner), and disable retries in inner SDKs, or the reverse; never both.
• Retry budgets: allow retries only while they are under ~10% of total requests per client (gRPC and service meshes support this); beyond that, fail fast.
• Circuit breakers stop calling a dependency that is failing consistently and probe it periodically.
• Propagate deadlines downstream (gRPC deadlines, X-Request-Deadline style headers) so inner services stop work the caller has abandoned.

Exactly-once, honestly

Networks give at-most-once or at-least-once. “Exactly-once” in practice means at-least-once delivery plus idempotent processing (dedupe by key, transactional outbox/inbox). Broker features such as Kafka transactions give exactly-once only within the broker’s own read-process-write loop, not for your database or external APIs.

Gotchas

• Generating the idempotency key inside the retry loop gives every attempt a new key: duplicates again.
• Client timeout shorter than server processing time causes retries while the first attempt still runs; the in-progress (409) path must exist.
• Retrying on 500 for a POST without a key creates duplicates.
• Fixed delays without jitter synchronize clients into waves (thundering herd).
• Background job frameworks retry by default; jobs must be idempotent or have retries disabled.
• Caching a 5xx as the idempotent result locks the client into a failure forever.

Verification

1. Send the same keyed request twice sequentially: one side effect, identical responses.
2. Send it twice concurrently: one succeeds, the other gets 409 or the replayed result.
3. Reuse the key with a different body: 422.
4. Kill the server mid-request, retry: the operation completes exactly once.
5. Inject 503s into a dependency: retries are bounded, jittered, and stop at the deadline.