Async Concurrency skill for Claude Code

Done well means: independent I/O runs concurrently, concurrency is bounded by a number you chose from the downstream’s capacity, every call has a timeout and a cancellation path, failures are handled per item without losing the rest, and throughput improves without raising the error rate of anything downstream.

Ground rules

• Concurrency is a resource budget, not a free speedup. Every concurrent request holds a socket, a pool connection, memory, and a slot in someone else’s rate limit.
• Never fan out unbounded: Promise.all(items.map(fetch)) over 10,000 items is a self-inflicted DDoS. Always cap.
• Every network call has a timeout. Missing timeouts turn a slow dependency into a full outage via exhausted pools.
• Retries multiply load. Retry only idempotent operations, with exponential backoff, jitter, a max attempt count, and respect for Retry-After.
• CPU-bound work does not benefit from async; it needs threads, processes, or workers.

Workflow

- [ ] 1. Map the calls: which are independent, which depend on earlier results
- [ ] 2. Find the binding limit: downstream rate limit, DB pool size, sockets, memory
- [ ] 3. Pick a concurrency cap from that limit (Little's law below)
- [ ] 4. Implement bounded concurrency with timeouts and per-item error handling
- [ ] 5. Add backpressure for producers faster than consumers
- [ ] 6. Load test; watch throughput, latency percentiles, downstream errors, pool waits

2–3. Choosing the number

Little’s law: concurrency = throughput × latency. To sustain 200 requests/s against a dependency with 50 ms p50 latency you need about 200 × 0.05 = 10 in flight; budget extra for tail latency (use p95, so maybe 20).

Binding limit	Cap
Third-party API with rate limit R req/s	Token bucket at R (or slightly under) and concurrency ≈ R × p95 latency
Database pool of P connections shared by the process	Concurrency per process ≤ P; total across replicas ≤ DB max_connections minus headroom
Your own downstream service	Agree a number; watch its CPU and p99
Memory per item (large payloads)	Cap so peak memory = cap × item size fits
Unknown	Start at 4–10, measure, increase until throughput stops improving or errors rise

DB pool sizing: more connections is not faster. A common starting point is around (2 × CPU cores of the DB server) active connections in total; beyond that, Postgres spends time context switching. Use a pooler (PgBouncer, RDS Proxy) when many app replicas share one database.

4. Bounded concurrency by language

TypeScript (p-limit, or a hand-rolled pool):

import pLimit from "p-limit";

const limit = pLimit(10);
const results = await Promise.allSettled(
  ids.map((id) => limit(() => fetchJson(`https://api.example.com/items/${id}`, { signal: AbortSignal.timeout(5_000) }))),
);
const ok = results.flatMap((r) => (r.status === "fulfilled" ? [r.value] : []));
const failed = results.flatMap((r, i) => (r.status === "rejected" ? [{ id: ids[i], error: String(r.reason) }] : []));

For very large inputs, do not create all promises up front (memory); process in a worker pool that pulls from an iterator:

async function mapPool<T, R>(items: Iterable<T>, size: number, fn: (item: T) => Promise<R>): Promise<R[]> {
  const it = items[Symbol.iterator]();
  const out: R[] = [];
  let index = 0;
  async function worker() {
    for (let next = it.next(); !next.done; next = it.next()) {
      const i = index++;
      out[i] = await fn(next.value);
    }
  }
  await Promise.all(Array.from({ length: size }, worker));
  return out;
}

Python asyncio (3.11+):

import asyncio, httpx

async def fetch_all(ids: list[str], limit: int = 10) -> dict[str, dict]:
    sem = asyncio.Semaphore(limit)
    results: dict[str, dict] = {}
    async with httpx.AsyncClient(timeout=httpx.Timeout(5.0),
                                 limits=httpx.Limits(max_connections=limit, max_keepalive_connections=limit)) as client:
        async def one(i: str) -> None:
            async with sem:
                r = await client.get(f"https://api.example.com/items/{i}")
                r.raise_for_status()
                results[i] = r.json()
        async with asyncio.TaskGroup() as tg:        # one failure cancels the rest
            for i in ids:
                tg.create_task(one(i))
    return results

Use asyncio.gather(..., return_exceptions=True) instead of TaskGroup when partial success is acceptable; inspect each result. asyncio.timeout(5) (3.11) wraps any block.

Go: errgroup.WithContext plus g.SetLimit(10) (see go-idioms for the full pattern); pass ctx to every request so the first failure cancels the rest.

5. Backpressure

When producers outpace consumers, something must slow down or drop:

Mechanism	Use
Bounded queue / channel; producer blocks when full	In-process pipelines
Streams with pipeline() (Node) / async iterators	File and network transforms; respects highWaterMark
Pull-based consumers (for await over a paginated source)	Importers, crawlers
Queue with consumer concurrency limit (SQS, RabbitMQ prefetch, BullMQ concurrency)	Cross-process work
Load shedding: return 429/503 with Retry-After when saturated	Servers protecting themselves

import { pipeline } from "node:stream/promises";
await pipeline(fs.createReadStream("big.ndjson"), parseNdjson(), transformBatch(500), writeToDb());

Never accumulate unbounded results in memory “to process later”; process and flush in batches.

Timeouts, cancellation, retries

• Set connect and total timeouts on every client (AbortSignal.timeout(ms) with fetch, httpx.Timeout, http.Client{Timeout} / context deadlines in Go).
• Propagate cancellation: when the caller gives up (client disconnects, parent task fails), cancel in-flight child calls via AbortController, task cancellation, or ctx.
• Overall deadline budget: if the request has 2 s total, give downstream calls less, not 2 s each.
• Retry policy:

async function withRetry<T>(fn: () => Promise<T>, attempts = 4, baseMs = 200): Promise<T> {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (err) {
      if (i >= attempts - 1 || !isRetryable(err)) throw err;          // 429, 502, 503, 504, network resets
      const delay = Math.random() * baseMs * 2 ** i;                    // full jitter
      await new Promise((r) => setTimeout(r, Math.min(delay, 10_000)));
    }
  }
}

Honor Retry-After when present. Use a retry budget or circuit breaker for hot paths so a failing dependency does not receive 4x traffic.

Keep-alive and pools

• Reuse clients: one httpx.AsyncClient, one undici Agent / global fetch, one Go http.Client per process. Creating a client per request defeats keep-alive and leaks sockets.
• Node’s global fetch (undici) pools connections per origin; tune with a custom Agent({ connections }) if a single origin needs more parallelism.
• Database: acquire a connection for the shortest possible time; never hold one while awaiting an HTTP call.

Gotchas

• Sequential awaits in a loop over independent calls (for (const id of ids) await get(id)) are the most common slowness; but switching to unbounded Promise.all is the most common overload.
• Array.forEach(async ...) does not await anything; errors become unhandled rejections.
• Nested pool acquisition (a task holding a DB connection waits for another connection) deadlocks when the pool is exhausted.
• Thundering herd: many instances retrying or refreshing a cache at the same moment. Add jitter to schedules and use request coalescing (single-flight).
• Rate limits are often per account across all your replicas; a per-process limiter of R with 5 replicas sends 5R.
• Async in Python with blocking libraries (requests, sync DB drivers) serializes everything on the event loop; use async clients or asyncio.to_thread.
• Ordering: concurrent results complete out of order; index results by input position or key when order matters.

Verification

1. Load test or run the batch on realistic volume: throughput up, p95/p99 latency acceptable, error rate unchanged.
2. Downstream metrics: no 429s or 5xx spikes, DB pool wait time near zero, connection counts within limits.
3. Kill or slow the dependency (fault injection, a mock with delays): timeouts fire, in-flight work cancels, the process stays healthy, memory stays bounded.
4. Partial failure test: one item fails; the rest complete and the failure is reported.

Report template

**Job:** sync 40k product prices from supplier API (limit 50 req/s, p95 latency 300 ms)
**Before:** sequential awaits, 3 h 40 min
**Change:** p-limit(15) + token bucket 45 req/s, 5 s timeout, 4 retries with full jitter on 429/5xx
**After:** 16 min; 0 rate-limit errors; 12 items failed permanently (listed in report.csv)
**Safety:** DB writes batched 500 per transaction; pool size 10 unchanged