Chaos Engineering skill for Claude Code

An experiment is done well when it starts from a measurable steady state, injects one fault with a bounded blast radius and an automatic abort, produces a yes/no answer to a written hypothesis, and ends with a ticket for every weakness found. Breaking things without a hypothesis is an outage you caused, not an experiment.

Hard rules

• No experiment without a written hypothesis, a steady-state metric, an abort condition and a named person who can stop it.
• Start in a non-production environment. Production experiments require explicit user approval, a small blast radius (one pod, one AZ slice, 1-5% of traffic), on-call awareness, and a time window outside peak and change freezes.
• Never run the first experiment of a new fault type in production.
• Every fault must have a duration limit and a tested rollback (delete the CR, stop the FIS experiment, remove the toxic, tc qdisc del).
• Do not inject faults into stateful data stores (disk fill, corrupting writes, killing the primary) until backups and restore were verified recently. See backup-restore.
• If monitoring cannot see the steady-state metric, fix observability first. A fault you cannot observe teaches nothing.

Workflow

- [ ] 1. Pick the question (which failure, which dependency, which promise)
- [ ] 2. Define steady state with a real metric and current baseline
- [ ] 3. Write the hypothesis and abort conditions
- [ ] 4. Choose tool and fault; scope the blast radius
- [ ] 5. Dry run in a lower environment; verify the rollback works
- [ ] 6. Run, observe, abort if conditions trip
- [ ] 7. Record results, file fixes, automate as a regression experiment

1. Pick the question

Derive candidates from real risk, not tool menus:

• Past incidents and near misses (postmortems list them).
• Dependencies on the critical path: database, cache, auth provider, payment API, DNS, message broker, object storage.
• Promises the team believes but has not tested: “we fail over in 60 s”, “retries cover transient errors”, “the cache going away only makes it slower”, “a node loss is invisible to users”.

Rank by (likelihood x impact) / effort. Good first experiments: kill one replica, add 200 ms latency to one downstream, make the cache unavailable, drain one node.

2. Steady state

Use a user-facing SLI, not CPU. Pull the baseline for the same hour on previous days.

# success ratio over 5m
sum(rate(http_requests_total{job="checkout",code!~"5.."}[5m]))
  / sum(rate(http_requests_total{job="checkout"}[5m]))

# p99 latency
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))

3. Hypothesis and abort conditions

Template:

**Hypothesis:** When 1 of 3 checkout pods is killed every 2 minutes for 10 minutes,
success ratio stays >= 99.5% and p99 stays < 800 ms (baseline 99.9% / 420 ms).
**Abort if:** success ratio < 99% for 2 consecutive minutes, p99 > 2 s, any page fires,
or a human says stop.
**Blast radius:** namespace shop-staging, label app=checkout, max 1 pod at a time.
**Rollback:** kubectl delete podchaos checkout-kill -n chaos (verified in dry run).
**Owner / stopper:** <name>; on-call notified in #ops at <time>.

4. Choose tool and fault

Target	Default tool	Alternatives
Kubernetes pods, network, IO, time, stress	Chaos Mesh	LitmusChaos (ChaosHub experiments, probes), Steadybit
AWS resources (EC2, ECS, EKS, RDS failover, AZ power interruption, API errors)	AWS Fault Injection Service with CloudWatch alarm stop conditions	Gremlin
A single TCP dependency in tests or local dev	Toxiproxy	tc netem on a test host
Linux host CPU/memory/IO pressure	stress-ng	Chaos Mesh StressChaos on k8s
Application-level faults (error rates, slow handlers)	Feature-flagged fault hooks in code	Service mesh fault injection (Istio fault, Linkerd)

Fault catalog by question:

Question	Fault
Does losing a replica affect users?	Pod kill / instance terminate
Do timeouts and retries work?	Latency (+200 ms, +2 s) and packet loss on one dependency
Does the app degrade when cache dies?	Block traffic to Redis (network partition)
Does failover meet RTO?	RDS/Aurora failover, zone outage simulation
Does autoscaling respond in time?	CPU stress or synthetic load spike
Are clocks assumed monotonic?	TimeChaos clock skew (non-prod only)
Is DNS a single point of failure?	DNS error injection for one domain

5-6. Run it

Chaos Mesh pod kill, scoped to one pod with a duration (Chaos Mesh v2.x CRDs):

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: checkout-kill
  namespace: chaos
spec:
  action: pod-kill
  mode: one                      # one | fixed | fixed-percent | random-max-percent | all
  selector:
    namespaces: [shop-staging]
    labelSelectors: { app: checkout }

Chaos Mesh latency on traffic from checkout to payments:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payments-latency
  namespace: chaos
spec:
  action: delay
  mode: all
  selector:
    namespaces: [shop-staging]
    labelSelectors: { app: checkout }
  direction: to
  target:
    mode: all
    selector:
      namespaces: [shop-staging]
      labelSelectors: { app: payments }
  delay: { latency: "200ms", jitter: "50ms", correlation: "25" }
  duration: "10m"

kubectl apply -f payments-latency.yaml
kubectl describe networkchaos payments-latency -n chaos   # check injection status per pod
kubectl delete networkchaos payments-latency -n chaos     # rollback / abort

Use a Schedule (or Workflow) object to repeat pod-kill every N minutes instead of re-applying by hand. Restrict Chaos Mesh with its namespace filter annotation so it can only touch namespaces explicitly enabled for chaos.

Toxiproxy in integration tests (deterministic, no cluster needed):

toxiproxy-server &
toxiproxy-cli create -l localhost:26379 -u localhost:6379 redis
toxiproxy-cli toxic add -t latency -a latency=1000 -a jitter=100 redis
toxiproxy-cli toxic add -t timeout -a timeout=0 redis     # hang connections
toxiproxy-cli toxic remove -n latency_downstream redis

Point the app at localhost:26379 and assert behavior in the test: timeouts fire, fallback served, error rate bounded, no thread or connection pool exhaustion.

Host-level network faults (test hosts only; requires root):

tc qdisc add dev eth0 root netem delay 200ms 50ms loss 1%
tc qdisc show dev eth0
tc qdisc del dev eth0 root netem      # always run this in a trap or timer
stress-ng --cpu 4 --cpu-load 80 --timeout 300s
stress-ng --vm 2 --vm-bytes 75% --timeout 120s

AWS FIS: create an experiment template with targets selected by tag, an action such as aws:ecs:stop-task, aws:ec2:stop-instances, aws:rds:failover-db-cluster or aws:fis:inject-api-throttle-error, and stopConditions pointing at CloudWatch alarms on the steady-state metric. Run with aws fis start-experiment --experiment-template-id <id> and stop with aws fis stop-experiment --id <id>. The FIS role must be scoped to tagged resources only.

During the run:

• Watch the steady-state dashboard and the abort metric live; one person drives, one observes, one takes timestamped notes.
• Abort immediately when a condition trips. Do not “give it another minute”.
• Confirm the fault actually applied (describe output, tc qdisc show, latency visible in traces). An experiment where the fault silently failed proves nothing.

7. Results and follow-up

## Experiment: payments latency +200 ms (staging, 2026-03-04 14:00-14:10 UTC)
**Hypothesis:** checkout success >= 99.5%, p99 < 800 ms. **Result:** REJECTED
**Observed:** success 97.1%; p99 2.4 s; connection pool to payments exhausted at 14:04.
**Why:** HTTP client timeout 30 s, no per-try timeout; 3 retries without backoff.
**Actions:** [OPS-412] per-try timeout 500 ms + budgeted retries; [OPS-413] pool metric alert.
**Rerun:** after OPS-412 ships; then add to weekly scheduled chaos in staging.

Turn every passed experiment into an automated recurring one (Chaos Mesh Schedule, FIS on a scheduler, Toxiproxy cases in CI) so regressions show up before production does.

Game days

• Announce scope, time, and abort rules a week ahead; invite on-call, service owners, and someone from support.
• Script 2-4 scenarios, ordered from least to most disruptive, each with its own hypothesis. Leave 30% of the time for unscripted follow-ups.
• Practice the human side: was the alert clear, did the runbook work, who was paged, how long until mitigation. Feed gaps into runbooks and incident-response.
• Hold a 30-minute debrief the same day; record findings in the template above.

Maturity path

1. Toxiproxy/fault tests in CI for each critical dependency.
2. Manual experiments in staging with Chaos Mesh/Litmus/FIS.
3. Scheduled recurring experiments in staging.
4. Small-blast-radius production experiments with alarm-based auto-stop, approved per run.
5. Continuous production chaos (only with mature SLO alerting and automated abort).

Gotchas

• Staging with one replica and no traffic tells you nothing about failover. Generate load (see load-testing) during experiments or results are vacuous.
• Retries hide faults in averages but explode p99 and downstream load. Always watch tail latency and dependency request rate, not just error ratio.
• Latency faults are more revealing than hard failures; most systems handle “connection refused” better than “slow”.
• tc and NetworkChaos on the wrong interface or direction can cut your own SSH or control-plane access. Use a timer-based cleanup and keep console access.
• Killing pods that have no PodDisruptionBudget or readiness probe tests the deployment config, not resilience. Fix basics first (see kubernetes-deploy).
• Chaos tooling has broad permissions. Restrict Chaos Mesh namespaces and FIS IAM roles to tagged resources.

Verification

• The fault was observed taking effect (tool status plus metric/trace evidence).
• Steady-state metrics returned to baseline after rollback; no lingering CRs, qdiscs, or FIS experiments (kubectl get podchaos,networkchaos -A, tc qdisc show).
• The report states hypothesis, result (confirmed/rejected), evidence, and linked tickets.