Resilience Patterns skill for Claude Code
One slow dependency should never sink the service. Claude starts with a dependency failure matrix that says what users see when each dependency is slow or down, then implements it: timeouts derived from latency budgets with deadline propagation, retry budgets that cannot amplify incidents, per-dependency breakers and bulkheads, load shedding before collapse, and liveness probes that never check the database. Fault injection proves each behavior before it is called done.
- Name
resilience-patterns- Size
- 190 lines · ≈2,450 tokens
- Files
- 2
- License
- MIT
- Updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Makes services survive slow and failing dependencies by building a per-dependency failure matrix and applying timeouts with deadline propagation, retry budgets, circuit breakers, bulkheads, load shedding and admission control, backpressure, graceful degradation with fallbacks, and correct liveness and readiness probes, configured in resilience4j, Polly, Envoy or Istio, opossum, gobreaker and Python equivalents, then proven with fault injection. Use when one slow dependency takes a whole service down, cascading failures or retry storms occur, adding a circuit breaker, or reviewing a design for failure handling. Try asking
- When the recommendations service is slow our whole checkout times out. Fix it
- Add circuit breakers and bulkheads to our outbound calls in this Spring service
- Review this design for cascading failure risks and propose concrete settings
Use it when
- A slow or failing dependency takes the whole service down
- Retry storms or cascading failures happened in an incident
- Adding timeouts, circuit breakers or bulkheads
- Reviewing a design or service for failure handling
Skip it when
- You need idempotency keys and per-call retry policy details — use idempotency-retries
- You are limiting clients' request rates — use rate-limiting
- You are setting availability targets and alerts — use slo-error-budgets
How to check the result
Fault injection of latency, errors and refused connections on each dependency produces the degraded behavior in the matrix within the latency budget, and excess load gets fast 503s instead of timeouts.
The common mistake it prevents
Adding retries at every layer without a budget, so a brief dependency hiccup becomes a self-inflicted outage as calls multiply on the way down.
Done well means: every outbound call has a timeout derived from a latency budget, retries are bounded by a budget so they cannot multiply load, each critical dependency is isolated so its failure consumes bounded resources, the service sheds excess load before it falls over, every dependency failure maps to a defined degraded behavior, and fault injection has proven all of it.
Workflow
- [ ] 1. Build the dependency failure matrix
- [ ] 2. Set the latency budget and derive timeouts; propagate deadlines
- [ ] 3. Bound retries with a retry budget (only for idempotent operations)
- [ ] 4. Add circuit breakers on remote dependencies that can fail as a whole
- [ ] 5. Isolate with bulkheads (separate pools/semaphores per dependency)
- [ ] 6. Add load shedding at the service entrance
- [ ] 7. Define degraded behavior and fallbacks per dependency
- [ ] 8. Fix health probes
- [ ] 9. Prove it with fault injection; add dashboards and alerts
1. Failure matrix
List every dependency the request path touches and decide behavior before writing code:
| Dependency | Criticality | p99 latency | Failure modes seen | If slow | If down | Degraded behavior |
|---|---|---|---|---|---|---|
| Postgres primary | Critical | 15 ms | Failover 30 s, connection exhaustion | Shed load | 503 with Retry-After | Read-only mode from replica |
| Payments API | Critical for checkout only | 400 ms | Timeouts, 5xx bursts | Breaker opens | Checkout disabled, cart works | “Payment delayed” queue |
| Recommendations | Optional | 120 ms | Slow under load | Skip after 100 ms | Skip | Empty section or cached list |
| Feature flags | Optional | 5 ms (local cache) | Outage | Serve cached | Serve last known / defaults | Defaults in code |
The matrix is the main artifact; the patterns below implement its columns.
2. Timeouts and deadlines
- Every network call gets a connect timeout and a total (request) timeout. Library defaults are often infinite or 30 to 100 seconds; never rely on them.
- Derive from budgets, not guesses: per-call timeout around the dependency’s p99.9 plus margin, and the sum along the critical path (including retries) must fit inside the caller’s own deadline.
- Propagate the remaining deadline downstream (gRPC deadlines do this natively; for HTTP, send a deadline or timeout header and have callees respect it). Work whose caller has already given up is wasted capacity.
- Database: statement timeout per role or per query (
SET statement_timeout), pool acquire timeout (fail in about 1 to 3 seconds instead of queueing forever). - Example budget: edge timeout 2 s, service budget 1.5 s, two sequential downstream calls at 500 ms each plus one retry of 300 ms on the second, leaving 200 ms headroom.
3. Retries with a budget
Retry only idempotent operations (or ones made idempotent with keys), only on transient errors (timeouts, 503, connection reset; never 4xx validation errors), with exponential backoff and full jitter, at one layer of the stack. Then cap retries globally:
- Retry budget: retries may add at most 10 to 20% extra load relative to first attempts; when exceeded, fail without retrying. Envoy and some client libraries support this directly; otherwise track a token bucket refilled per successful first attempt.
- Three layers each retrying 3 times turns one user request into 27 calls at the bottom. Choose the one layer that retries (usually the closest to the failing call) and set the others to zero.
The detailed retry policy (which errors, backoff formula, Retry-After, idempotency keys) belongs to idempotency-retries; this skill only bounds retries system-wide.
4. Circuit breakers
Use a breaker when a dependency can fail as a whole and calling it while it is failing costs you threads, connections or latency.
| Setting | Starting value | Notes |
|---|---|---|
| Sliding window | Last 50 to 100 calls, or 30 to 60 s | Count-based for steady traffic, time-based for bursty |
| Minimum calls before evaluating | 20 | Prevents opening on 2 failures out of 3 |
| Failure-rate threshold | 50% | Count timeouts and 5xx; never count 4xx client errors |
| Slow-call threshold | Calls slower than the timeout budget at 50%+ | Slow is the more common failure than errors |
| Open duration | 15 to 60 s | Then half-open |
| Half-open probe calls | 3 to 10 | Close if they succeed; reopen otherwise |
- One breaker per dependency (and per endpoint when endpoints fail independently), not one global breaker.
- When open, fail fast into the degraded behavior from the matrix. A breaker without a defined fallback just produces faster errors.
- Emit state changes as metrics and logs; alert when a breaker stays open.
- Service meshes provide outlier detection (eject one bad instance) which complements, but does not replace, an application breaker for a whole dependency outage.
5. Bulkheads
Isolate resources so one slow dependency cannot consume everything:
- Separate connection pools or HTTP clients per dependency, each with a max size.
- A concurrency limit (semaphore) per dependency: for example payments max 20 in flight; extra calls fail immediately or wait at most a few milliseconds.
- Separate worker pools or queues for critical and non-critical work (checkout vs report generation); separate deployments for traffic classes when needed.
- Size from Little’s law: concurrency = throughput x latency. 100 req/s to a dependency at 200 ms p99 needs about 20 concurrent slots; set the limit slightly above, not 10x above.
6. Load shedding and admission control
A service at capacity should reject quickly rather than queue until everything times out.
- Limit concurrent in-flight requests per instance; return 503 (or 429 for per-client
limits) with
Retry-Afterwhen exceeded. - Bound every queue and drop requests that have waited longer than the client’s timeout (they are already abandoned). Prefer serving newest requests under overload.
- Shed by priority: health checks and critical paths first, background and bulk traffic last (a request criticality header or separate routes).
- Adaptive concurrency limits (Envoy adaptive concurrency filter, Netflix concurrency-limits) adjust the limit from observed latency; use them when traffic and capacity change often.
- Clients must back off on 503/429; otherwise shedding just triggers retry storms.
7. Degradation and fallbacks
| Fallback | Use when | Caution |
|---|---|---|
| Serve stale cache | Data tolerates staleness | Mark responses as stale where users care |
| Default value | Feature flags, config, recommendations | Defaults must be safe (feature off, not on) |
| Skip the section | Optional UI parts | Return explicit empty state, not an error |
| Queue for later | Writes that can be async (emails, analytics, some payments) | Needs durable queue and idempotent replay |
| Read-only mode | Primary database down, replicas up | Disable writes clearly in UI and API |
| Static response | Total outage of a non-critical page | Keep it truly static (CDN) |
Never fall back to a path that is less safe: an authorization check that fails must deny, not allow.
8. Health probes
- Liveness: “this process is wedged and needs a restart”. Check only in-process health (event loop responsive, not deadlocked). Never check databases or downstream services in liveness; a dependency outage would restart every pod and make things worse.
- Readiness: “send me traffic”. May check critical dependencies with short timeouts and should fail during startup, shutdown drain, and when shedding everything.
- Startup probe for slow-booting apps so liveness does not kill them during init.
9. Prove it
Fault injection in staging (or production with guardrails and explicit approval):
| Fault | Tool | Expected |
|---|---|---|
| Dependency latency +2 s | toxiproxy, Envoy fault filter, tc netem delay | Timeouts fire, breaker opens, degraded response within budget |
| Dependency 100% 5xx | Envoy/Istio fault abort, stub server | Breaker opens within window, fallback served |
| Connection refused | Stop the dependency container | Fast failure, no thread or connection pile-up |
| Traffic 3x capacity | k6 arrival-rate test | 503s for the excess, p99 of accepted requests stays near normal |
| Pod kill during load | Chaos Mesh, kubectl delete pod | No user-visible errors beyond in-flight retries |
Watch: thread/connection pool saturation, queue lengths, breaker state, retry rate versus budget, and p99 latency of accepted requests. Chaos experiments in production require explicit user approval, a blast-radius limit, and an abort switch.
Library and proxy configurations for each pattern (resilience4j, Polly v8 and Microsoft.Extensions.Http.Resilience, Envoy and Istio, opossum, gobreaker, Python) are in references/library-configs.md; read it when implementing in a specific stack.
Gotchas
- Timeouts longer than the caller’s timeout do nothing except waste resources.
- A breaker that counts 404 or validation errors as failures opens on user mistakes.
- Shared HTTP client pools across dependencies defeat bulkheads silently.
- Retrying on the server side and the client side doubles load during the incident you are trying to survive.
- Fallbacks that are never exercised rot; inject the failure regularly.
- Health checks that call the database every second from every pod become a load source.
Output contract
## Dependency failure matrix
(table as in step 1)
## Budgets and timeouts
| Call | Timeout | Retries | Deadline propagated |
## Breakers and bulkheads
| Dependency | Breaker settings | Concurrency limit | Fallback |
## Load shedding
Limits, priorities, client backoff contract
## Probes
## Fault injection results
| Fault | Expected | Observed | Action | Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
references/library-configs.md
# Resilience configurations by stack
Starting configurations implementing timeouts, breakers, bulkheads and retry budgets.
Tune values from the dependency failure matrix; confirm option names against the
installed library version.
## Java: resilience4j (Spring Boot)
```yaml
resilience4j:
circuitbreaker:
instances:
payments:
slidingWindowType: COUNT_BASED
slidingWindowSize: 50
minimumNumberOfCalls: 20
failureRateThreshold: 50
slowCallDurationThreshold: 800ms
slowCallRateThreshold: 50
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 5
automaticTransitionFromOpenToHalfOpenEnabled: true
ignoreExceptions:
- com.acme.payments.PaymentDeclinedException # business errors are not failures
timelimiter:
instances:
payments:
timeoutDuration: 1s
bulkhead:
instances:
payments:
maxConcurrentCalls: 20
maxWaitDuration: 5ms
retry:
instances:
payments:
maxAttempts: 2 # 1 retry
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
retryExceptions:
- java.net.SocketTimeoutException
- java.io.IOException
```
```java
@CircuitBreaker(name = "payments", fallbackMethod = "queuePayment")
@Bulkhead(name = "payments")
@Retry(name = "payments")
public PaymentResult charge(ChargeRequest req) { return client.charge(req); }
private PaymentResult queuePayment(ChargeRequest req, Throwable t) {
outbox.enqueue(req); // degraded: process later
return PaymentResult.pending();
}
```
Aspect order matters (by default Retry wraps CircuitBreaker wraps Bulkhead...); check the
configured aspect order so retries do not bypass the breaker. Also set HTTP client
connect and read timeouts; `timelimiter` applies only to async return types.
## .NET: Polly v8 and Microsoft.Extensions.Http.Resilience
Default for HttpClient:
```csharp
builder.Services.AddHttpClient<PaymentsClient>(c => c.BaseAddress = new("https://payments.internal"))
.AddStandardResilienceHandler(o =>
{
o.AttemptTimeout.Timeout = TimeSpan.FromMilliseconds(800);
o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(2);
o.Retry.MaxRetryAttempts = 1;
o.CircuitBreaker.FailureRatio = 0.5;
o.CircuitBreaker.MinimumThroughput = 20;
o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);
});
```
Custom pipeline for non-HTTP calls:
```csharp
var pipeline = new ResiliencePipelineBuilder<Quote>()
.AddFallback(new() { FallbackAction = _ => Outcome.FromResultAsValueTask(Quote.Cached()) })
.AddCircuitBreaker(new() { FailureRatio = 0.5, MinimumThroughput = 20,
SamplingDuration = TimeSpan.FromSeconds(30), BreakDuration = TimeSpan.FromSeconds(30) })
.AddConcurrencyLimiter(permitLimit: 20, queueLimit: 0) // bulkhead (Polly.RateLimiting)
.AddTimeout(TimeSpan.FromMilliseconds(800))
.Build();
var quote = await pipeline.ExecuteAsync(async ct => await pricing.GetQuoteAsync(id, ct), ct);
```
Strategies execute outermost first in the order added. ASP.NET Core's built-in
`AddRateLimiter` with a `ConcurrencyLimiter` policy provides inbound load shedding.
## Envoy
```yaml
clusters:
- name: payments
connect_timeout: 0.25s
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 100
max_pending_requests: 50 # bulkhead on queued requests
max_requests: 200
retry_budget:
budget_percent: { value: 20.0 } # retries <= 20% of active requests
min_retry_concurrency: 3
outlier_detection:
consecutive_5xx: 5
interval: 10s
base_ejection_time: 30s
max_ejection_percent: 50
routes:
- match: { prefix: "/payments" }
route:
cluster: payments
timeout: 2s
retry_policy:
retry_on: "connect-failure,reset,retriable-status-codes"
retriable_status_codes: [503]
num_retries: 1
per_try_timeout: 0.8s
```
Envoy's adaptive concurrency HTTP filter can shed load based on measured latency; enable
it only after the static limits work.
## Istio
```yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: payments }
spec:
host: payments.prod.svc.cluster.local
trafficPolicy:
connectionPool:
tcp: { maxConnections: 100, connectTimeout: 250ms }
http: { http1MaxPendingRequests: 50, http2MaxRequests: 200, maxRetries: 3 }
outlierDetection:
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payments }
spec:
hosts: [payments.prod.svc.cluster.local]
http:
- timeout: 2s
retries: { attempts: 1, perTryTimeout: 800ms, retryOn: "connect-failure,reset,503" }
route: [{ destination: { host: payments.prod.svc.cluster.local } }]
```
Fault injection for tests: add `fault: { delay: { percentage: { value: 50 }, fixedDelay: 2s } }`
or `abort: { httpStatus: 503, percentage: { value: 100 } }` to a test-only route.
## Node.js: opossum, AbortSignal, p-limit
```ts
import CircuitBreaker from "opossum";
import pLimit from "p-limit";
const limit = pLimit(20); // bulkhead: 20 in flight
async function callPricing(id: string) {
const res = await fetch(`${PRICING_URL}/quotes/${id}`, { signal: AbortSignal.timeout(800) });
if (res.status >= 500) throw new Error(`pricing ${res.status}`);
if (!res.ok) return { error: res.status }; // 4xx: not a breaker failure
return res.json();
}
const breaker = new CircuitBreaker((id: string) => limit(() => callPricing(id)), {
timeout: 1000, // breaker-level timeout (slightly above fetch timeout)
errorThresholdPercentage: 50,
volumeThreshold: 20, // minimum calls before tripping
resetTimeout: 30_000, // open duration before half-open
rollingCountTimeout: 30_000,
});
breaker.fallback((id: string) => quoteCache.get(id) ?? null);
breaker.on("open", () => metrics.increment("breaker.pricing.open"));
const quote = await breaker.fire(id);
```
## Go: context deadlines, gobreaker, semaphores
```go
var cb = gobreaker.NewCircuitBreaker[*Quote](gobreaker.Settings{
Name: "pricing",
MaxRequests: 5, // half-open probes
Interval: 30 * time.Second, // closed-state counter reset
Timeout: 30 * time.Second, // open duration
ReadyToTrip: func(c gobreaker.Counts) bool {
return c.Requests >= 20 && float64(c.TotalFailures)/float64(c.Requests) >= 0.5
},
})
var sem = semaphore.NewWeighted(20) // golang.org/x/sync/semaphore
func GetQuote(ctx context.Context, id string) (*Quote, error) {
if !sem.TryAcquire(1) {
return cachedQuote(id), nil // bulkhead full: degrade immediately
}
defer sem.Release(1)
ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
defer cancel()
q, err := cb.Execute(func() (*Quote, error) { return pricing.Get(ctx, id) })
if errors.Is(err, gobreaker.ErrOpenState) || errors.Is(err, gobreaker.ErrTooManyRequests) {
return cachedQuote(id), nil
}
return q, err
}
```
Set `http.Client{Timeout: ...}` and transport `MaxConnsPerHost` per dependency client.
This uses gobreaker v2 (generic API, module path `github.com/sony/gobreaker/v2`).
## Python: httpx timeouts, pybreaker, semaphores
```python
import threading
import httpx
import pybreaker
client = httpx.Client(base_url=PRICING_URL,
timeout=httpx.Timeout(0.8, connect=0.25),
limits=httpx.Limits(max_connections=20)) # pool size doubles as a bulkhead
breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30,
exclude=[lambda e: isinstance(e, httpx.HTTPStatusError)
and e.response.status_code < 500])
bulkhead = threading.BoundedSemaphore(20)
@breaker
def _fetch_quote(qid: str) -> dict:
resp = client.get(f"/quotes/{qid}")
resp.raise_for_status()
return resp.json()
def get_quote(qid: str) -> dict | None:
if not bulkhead.acquire(blocking=False):
return cached_quote(qid) # bulkhead full: degrade immediately
try:
return _fetch_quote(qid)
except (pybreaker.CircuitBreakerError, httpx.TransportError):
return cached_quote(qid)
finally:
bulkhead.release()
```
- pybreaker opens after `fail_max` consecutive failures rather than a failure rate over a
window; for rate-based breaking, pick a library with sliding windows or wrap your own
counter.
- asyncio services: bound concurrency with `asyncio.Semaphore`, time out with
`async with asyncio.timeout(1.0)` (Python 3.11+), and use an asyncio-native breaker
library (verify its API) rather than calling a thread-based breaker from the event loop.
- Retries: tenacity with `stop=stop_after_attempt(2)` and
`wait=wait_random_exponential(multiplier=0.1, max=1)`, retrying only transport errors
and 503.
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.