Architecture Editor's pick

Resilience Patterns skill for Claude Code

One slow dependency should never sink the service. Claude starts with a dependency failure matrix that says what users see when each dependency is slow or down, then implements it: timeouts derived from latency budgets with deadline propagation, retry budgets that cannot amplify incidents, per-dependency breakers and bulkheads, load shedding before collapse, and liveness probes that never check the database. Fault injection proves each behavior before it is called done.

Name
resilience-patterns
Size
190 lines · ≈2,450 tokens
Files
2
License
MIT
Updated

What Claude sees before loading it

This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.

description: Makes services survive slow and failing dependencies by building a per-dependency failure matrix and applying timeouts with deadline propagation, retry budgets, circuit breakers, bulkheads, load shedding and admission control, backpressure, graceful degradation with fallbacks, and correct liveness and readiness probes, configured in resilience4j, Polly, Envoy or Istio, opossum, gobreaker and Python equivalents, then proven with fault injection. Use when one slow dependency takes a whole service down, cascading failures or retry storms occur, adding a circuit breaker, or reviewing a design for failure handling.

Try asking

  • When the recommendations service is slow our whole checkout times out. Fix it
  • Add circuit breakers and bulkheads to our outbound calls in this Spring service
  • Review this design for cascading failure risks and propose concrete settings

Use it when

  • A slow or failing dependency takes the whole service down
  • Retry storms or cascading failures happened in an incident
  • Adding timeouts, circuit breakers or bulkheads
  • Reviewing a design or service for failure handling

Skip it when

How to check the result

Fault injection of latency, errors and refused connections on each dependency produces the degraded behavior in the matrix within the latency budget, and excess load gets fast 503s instead of timeouts.

The common mistake it prevents

Adding retries at every layer without a budget, so a brief dependency hiccup becomes a self-inflicted outage as calls multiply on the way down.

~/.claude/skills/resilience-patterns/SKILL.md

--- name: resilience-patterns description: Makes services survive slow and failing dependencies by building a per-dependency failure matrix and applying timeouts with deadline… license: MIT ---

Done well means: every outbound call has a timeout derived from a latency budget, retries are bounded by a budget so they cannot multiply load, each critical dependency is isolated so its failure consumes bounded resources, the service sheds excess load before it falls over, every dependency failure maps to a defined degraded behavior, and fault injection has proven all of it.

Workflow

- [ ] 1. Build the dependency failure matrix
- [ ] 2. Set the latency budget and derive timeouts; propagate deadlines
- [ ] 3. Bound retries with a retry budget (only for idempotent operations)
- [ ] 4. Add circuit breakers on remote dependencies that can fail as a whole
- [ ] 5. Isolate with bulkheads (separate pools/semaphores per dependency)
- [ ] 6. Add load shedding at the service entrance
- [ ] 7. Define degraded behavior and fallbacks per dependency
- [ ] 8. Fix health probes
- [ ] 9. Prove it with fault injection; add dashboards and alerts

1. Failure matrix

List every dependency the request path touches and decide behavior before writing code:

DependencyCriticalityp99 latencyFailure modes seenIf slowIf downDegraded behavior
Postgres primaryCritical15 msFailover 30 s, connection exhaustionShed load503 with Retry-AfterRead-only mode from replica
Payments APICritical for checkout only400 msTimeouts, 5xx burstsBreaker opensCheckout disabled, cart works“Payment delayed” queue
RecommendationsOptional120 msSlow under loadSkip after 100 msSkipEmpty section or cached list
Feature flagsOptional5 ms (local cache)OutageServe cachedServe last known / defaultsDefaults in code

The matrix is the main artifact; the patterns below implement its columns.

2. Timeouts and deadlines

  • Every network call gets a connect timeout and a total (request) timeout. Library defaults are often infinite or 30 to 100 seconds; never rely on them.
  • Derive from budgets, not guesses: per-call timeout around the dependency’s p99.9 plus margin, and the sum along the critical path (including retries) must fit inside the caller’s own deadline.
  • Propagate the remaining deadline downstream (gRPC deadlines do this natively; for HTTP, send a deadline or timeout header and have callees respect it). Work whose caller has already given up is wasted capacity.
  • Database: statement timeout per role or per query (SET statement_timeout), pool acquire timeout (fail in about 1 to 3 seconds instead of queueing forever).
  • Example budget: edge timeout 2 s, service budget 1.5 s, two sequential downstream calls at 500 ms each plus one retry of 300 ms on the second, leaving 200 ms headroom.

3. Retries with a budget

Retry only idempotent operations (or ones made idempotent with keys), only on transient errors (timeouts, 503, connection reset; never 4xx validation errors), with exponential backoff and full jitter, at one layer of the stack. Then cap retries globally:

  • Retry budget: retries may add at most 10 to 20% extra load relative to first attempts; when exceeded, fail without retrying. Envoy and some client libraries support this directly; otherwise track a token bucket refilled per successful first attempt.
  • Three layers each retrying 3 times turns one user request into 27 calls at the bottom. Choose the one layer that retries (usually the closest to the failing call) and set the others to zero.

The detailed retry policy (which errors, backoff formula, Retry-After, idempotency keys) belongs to idempotency-retries; this skill only bounds retries system-wide.

4. Circuit breakers

Use a breaker when a dependency can fail as a whole and calling it while it is failing costs you threads, connections or latency.

SettingStarting valueNotes
Sliding windowLast 50 to 100 calls, or 30 to 60 sCount-based for steady traffic, time-based for bursty
Minimum calls before evaluating20Prevents opening on 2 failures out of 3
Failure-rate threshold50%Count timeouts and 5xx; never count 4xx client errors
Slow-call thresholdCalls slower than the timeout budget at 50%+Slow is the more common failure than errors
Open duration15 to 60 sThen half-open
Half-open probe calls3 to 10Close if they succeed; reopen otherwise
  • One breaker per dependency (and per endpoint when endpoints fail independently), not one global breaker.
  • When open, fail fast into the degraded behavior from the matrix. A breaker without a defined fallback just produces faster errors.
  • Emit state changes as metrics and logs; alert when a breaker stays open.
  • Service meshes provide outlier detection (eject one bad instance) which complements, but does not replace, an application breaker for a whole dependency outage.

5. Bulkheads

Isolate resources so one slow dependency cannot consume everything:

  • Separate connection pools or HTTP clients per dependency, each with a max size.
  • A concurrency limit (semaphore) per dependency: for example payments max 20 in flight; extra calls fail immediately or wait at most a few milliseconds.
  • Separate worker pools or queues for critical and non-critical work (checkout vs report generation); separate deployments for traffic classes when needed.
  • Size from Little’s law: concurrency = throughput x latency. 100 req/s to a dependency at 200 ms p99 needs about 20 concurrent slots; set the limit slightly above, not 10x above.

6. Load shedding and admission control

A service at capacity should reject quickly rather than queue until everything times out.

  • Limit concurrent in-flight requests per instance; return 503 (or 429 for per-client limits) with Retry-After when exceeded.
  • Bound every queue and drop requests that have waited longer than the client’s timeout (they are already abandoned). Prefer serving newest requests under overload.
  • Shed by priority: health checks and critical paths first, background and bulk traffic last (a request criticality header or separate routes).
  • Adaptive concurrency limits (Envoy adaptive concurrency filter, Netflix concurrency-limits) adjust the limit from observed latency; use them when traffic and capacity change often.
  • Clients must back off on 503/429; otherwise shedding just triggers retry storms.

7. Degradation and fallbacks

FallbackUse whenCaution
Serve stale cacheData tolerates stalenessMark responses as stale where users care
Default valueFeature flags, config, recommendationsDefaults must be safe (feature off, not on)
Skip the sectionOptional UI partsReturn explicit empty state, not an error
Queue for laterWrites that can be async (emails, analytics, some payments)Needs durable queue and idempotent replay
Read-only modePrimary database down, replicas upDisable writes clearly in UI and API
Static responseTotal outage of a non-critical pageKeep it truly static (CDN)

Never fall back to a path that is less safe: an authorization check that fails must deny, not allow.

8. Health probes

  • Liveness: “this process is wedged and needs a restart”. Check only in-process health (event loop responsive, not deadlocked). Never check databases or downstream services in liveness; a dependency outage would restart every pod and make things worse.
  • Readiness: “send me traffic”. May check critical dependencies with short timeouts and should fail during startup, shutdown drain, and when shedding everything.
  • Startup probe for slow-booting apps so liveness does not kill them during init.

9. Prove it

Fault injection in staging (or production with guardrails and explicit approval):

FaultToolExpected
Dependency latency +2 stoxiproxy, Envoy fault filter, tc netem delayTimeouts fire, breaker opens, degraded response within budget
Dependency 100% 5xxEnvoy/Istio fault abort, stub serverBreaker opens within window, fallback served
Connection refusedStop the dependency containerFast failure, no thread or connection pile-up
Traffic 3x capacityk6 arrival-rate test503s for the excess, p99 of accepted requests stays near normal
Pod kill during loadChaos Mesh, kubectl delete podNo user-visible errors beyond in-flight retries

Watch: thread/connection pool saturation, queue lengths, breaker state, retry rate versus budget, and p99 latency of accepted requests. Chaos experiments in production require explicit user approval, a blast-radius limit, and an abort switch.

Library and proxy configurations for each pattern (resilience4j, Polly v8 and Microsoft.Extensions.Http.Resilience, Envoy and Istio, opossum, gobreaker, Python) are in references/library-configs.md; read it when implementing in a specific stack.

Gotchas

  • Timeouts longer than the caller’s timeout do nothing except waste resources.
  • A breaker that counts 404 or validation errors as failures opens on user mistakes.
  • Shared HTTP client pools across dependencies defeat bulkheads silently.
  • Retrying on the server side and the client side doubles load during the incident you are trying to survive.
  • Fallbacks that are never exercised rot; inject the failure regularly.
  • Health checks that call the database every second from every pod become a load source.

Output contract

## Dependency failure matrix
(table as in step 1)
## Budgets and timeouts
| Call | Timeout | Retries | Deadline propagated |
## Breakers and bulkheads
| Dependency | Breaker settings | Concurrency limit | Fallback |
## Load shedding
Limits, priorities, client backoff contract
## Probes
## Fault injection results
| Fault | Expected | Observed | Action |

Bundled files

Loaded only when a step in the skill needs them. Review them here before installing.

references/library-configs.md 264 lines · 9.0 KB
# Resilience configurations by stack

Starting configurations implementing timeouts, breakers, bulkheads and retry budgets.
Tune values from the dependency failure matrix; confirm option names against the
installed library version.

## Java: resilience4j (Spring Boot)

```yaml
resilience4j:
  circuitbreaker:
    instances:
      payments:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 50
        minimumNumberOfCalls: 20
        failureRateThreshold: 50
        slowCallDurationThreshold: 800ms
        slowCallRateThreshold: 50
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        automaticTransitionFromOpenToHalfOpenEnabled: true
        ignoreExceptions:
          - com.acme.payments.PaymentDeclinedException   # business errors are not failures
  timelimiter:
    instances:
      payments:
        timeoutDuration: 1s
  bulkhead:
    instances:
      payments:
        maxConcurrentCalls: 20
        maxWaitDuration: 5ms
  retry:
    instances:
      payments:
        maxAttempts: 2                     # 1 retry
        waitDuration: 100ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - java.net.SocketTimeoutException
          - java.io.IOException
```

```java
@CircuitBreaker(name = "payments", fallbackMethod = "queuePayment")
@Bulkhead(name = "payments")
@Retry(name = "payments")
public PaymentResult charge(ChargeRequest req) { return client.charge(req); }

private PaymentResult queuePayment(ChargeRequest req, Throwable t) {
    outbox.enqueue(req);                   // degraded: process later
    return PaymentResult.pending();
}
```

Aspect order matters (by default Retry wraps CircuitBreaker wraps Bulkhead...); check the
configured aspect order so retries do not bypass the breaker. Also set HTTP client
connect and read timeouts; `timelimiter` applies only to async return types.

## .NET: Polly v8 and Microsoft.Extensions.Http.Resilience

Default for HttpClient:

```csharp
builder.Services.AddHttpClient<PaymentsClient>(c => c.BaseAddress = new("https://payments.internal"))
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout = TimeSpan.FromMilliseconds(800);
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(2);
        o.Retry.MaxRetryAttempts = 1;
        o.CircuitBreaker.FailureRatio = 0.5;
        o.CircuitBreaker.MinimumThroughput = 20;
        o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
        o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
    });
```

Custom pipeline for non-HTTP calls:

```csharp
var pipeline = new ResiliencePipelineBuilder<Quote>()
    .AddFallback(new() { FallbackAction = _ => Outcome.FromResultAsValueTask(Quote.Cached()) })
    .AddCircuitBreaker(new() { FailureRatio = 0.5, MinimumThroughput = 20,
        SamplingDuration = TimeSpan.FromSeconds(30), BreakDuration = TimeSpan.FromSeconds(30) })
    .AddConcurrencyLimiter(permitLimit: 20, queueLimit: 0)     // bulkhead (Polly.RateLimiting)
    .AddTimeout(TimeSpan.FromMilliseconds(800))
    .Build();

var quote = await pipeline.ExecuteAsync(async ct => await pricing.GetQuoteAsync(id, ct), ct);
```

Strategies execute outermost first in the order added. ASP.NET Core's built-in
`AddRateLimiter` with a `ConcurrencyLimiter` policy provides inbound load shedding.

## Envoy

```yaml
clusters:
- name: payments
  connect_timeout: 0.25s
  circuit_breakers:
    thresholds:
    - priority: DEFAULT
      max_connections: 100
      max_pending_requests: 50          # bulkhead on queued requests
      max_requests: 200
      retry_budget:
        budget_percent: { value: 20.0 } # retries <= 20% of active requests
        min_retry_concurrency: 3
  outlier_detection:
    consecutive_5xx: 5
    interval: 10s
    base_ejection_time: 30s
    max_ejection_percent: 50
routes:
- match: { prefix: "/payments" }
  route:
    cluster: payments
    timeout: 2s
    retry_policy:
      retry_on: "connect-failure,reset,retriable-status-codes"
      retriable_status_codes: [503]
      num_retries: 1
      per_try_timeout: 0.8s
```

Envoy's adaptive concurrency HTTP filter can shed load based on measured latency; enable
it only after the static limits work.

## Istio

```yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: payments }
spec:
  host: payments.prod.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp: { maxConnections: 100, connectTimeout: 250ms }
      http: { http1MaxPendingRequests: 50, http2MaxRequests: 200, maxRetries: 3 }
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payments }
spec:
  hosts: [payments.prod.svc.cluster.local]
  http:
  - timeout: 2s
    retries: { attempts: 1, perTryTimeout: 800ms, retryOn: "connect-failure,reset,503" }
    route: [{ destination: { host: payments.prod.svc.cluster.local } }]
```

Fault injection for tests: add `fault: { delay: { percentage: { value: 50 }, fixedDelay: 2s } }`
or `abort: { httpStatus: 503, percentage: { value: 100 } }` to a test-only route.

## Node.js: opossum, AbortSignal, p-limit

```ts
import CircuitBreaker from "opossum";
import pLimit from "p-limit";

const limit = pLimit(20);                                   // bulkhead: 20 in flight

async function callPricing(id: string) {
  const res = await fetch(`${PRICING_URL}/quotes/${id}`, { signal: AbortSignal.timeout(800) });
  if (res.status >= 500) throw new Error(`pricing ${res.status}`);
  if (!res.ok) return { error: res.status };                // 4xx: not a breaker failure
  return res.json();
}

const breaker = new CircuitBreaker((id: string) => limit(() => callPricing(id)), {
  timeout: 1000,                   // breaker-level timeout (slightly above fetch timeout)
  errorThresholdPercentage: 50,
  volumeThreshold: 20,             // minimum calls before tripping
  resetTimeout: 30_000,            // open duration before half-open
  rollingCountTimeout: 30_000,
});
breaker.fallback((id: string) => quoteCache.get(id) ?? null);
breaker.on("open", () => metrics.increment("breaker.pricing.open"));

const quote = await breaker.fire(id);
```

## Go: context deadlines, gobreaker, semaphores

```go
var cb = gobreaker.NewCircuitBreaker[*Quote](gobreaker.Settings{
    Name:        "pricing",
    MaxRequests: 5,                       // half-open probes
    Interval:    30 * time.Second,        // closed-state counter reset
    Timeout:     30 * time.Second,        // open duration
    ReadyToTrip: func(c gobreaker.Counts) bool {
        return c.Requests >= 20 && float64(c.TotalFailures)/float64(c.Requests) >= 0.5
    },
})
var sem = semaphore.NewWeighted(20)     // golang.org/x/sync/semaphore

func GetQuote(ctx context.Context, id string) (*Quote, error) {
    if !sem.TryAcquire(1) {
        return cachedQuote(id), nil         // bulkhead full: degrade immediately
    }
    defer sem.Release(1)
    ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
    defer cancel()
    q, err := cb.Execute(func() (*Quote, error) { return pricing.Get(ctx, id) })
    if errors.Is(err, gobreaker.ErrOpenState) || errors.Is(err, gobreaker.ErrTooManyRequests) {
        return cachedQuote(id), nil
    }
    return q, err
}
```

Set `http.Client{Timeout: ...}` and transport `MaxConnsPerHost` per dependency client.
This uses gobreaker v2 (generic API, module path `github.com/sony/gobreaker/v2`).

## Python: httpx timeouts, pybreaker, semaphores

```python
import threading
import httpx
import pybreaker

client = httpx.Client(base_url=PRICING_URL,
                      timeout=httpx.Timeout(0.8, connect=0.25),
                      limits=httpx.Limits(max_connections=20))   # pool size doubles as a bulkhead
breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30,
                                   exclude=[lambda e: isinstance(e, httpx.HTTPStatusError)
                                            and e.response.status_code < 500])
bulkhead = threading.BoundedSemaphore(20)

@breaker
def _fetch_quote(qid: str) -> dict:
    resp = client.get(f"/quotes/{qid}")
    resp.raise_for_status()
    return resp.json()

def get_quote(qid: str) -> dict | None:
    if not bulkhead.acquire(blocking=False):
        return cached_quote(qid)                    # bulkhead full: degrade immediately
    try:
        return _fetch_quote(qid)
    except (pybreaker.CircuitBreakerError, httpx.TransportError):
        return cached_quote(qid)
    finally:
        bulkhead.release()
```

- pybreaker opens after `fail_max` consecutive failures rather than a failure rate over a
  window; for rate-based breaking, pick a library with sliding windows or wrap your own
  counter.
- asyncio services: bound concurrency with `asyncio.Semaphore`, time out with
  `async with asyncio.timeout(1.0)` (Python 3.11+), and use an asyncio-native breaker
  library (verify its API) rather than calling a thread-based breaker from the event loop.
- Retries: tenacity with `stop=stop_after_attempt(2)` and
  `wait=wait_random_exponential(multiplier=0.1, max=1)`, retrying only transport errors
  and 503.

About this skill

Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.

esc