Incident Response skill for Claude Code

During an incident the goal is to restore service, not to understand it. Done well means: impact is stopped quickly with a reversible action, everyone knows who is in charge, stakeholders get updates on a predictable cadence, and a timeline exists for the postmortem without anyone reconstructing it from memory.

Ground rules

• Mitigate before you diagnose. A rollback you can explain later beats a fix you are still writing.
• Prefer reversible actions (rollback, flag off, scale out, drain a region) over irreversible ones (data deletion, schema changes, cache flushes of primary data).
• Every production-changing action is proposed with its expected effect and rollback, then executed only after explicit confirmation from the incident commander/user.
• One change at a time; watch the key metric for its effect before the next.
• Write everything down with UTC timestamps as it happens.
• Never speculate about cause in external comms. State impact, actions, next update time.

Severity

Sev	Definition	Response	Update cadence
SEV1	Core service down or data loss/corruption or security breach, many customers	All hands, IC paged, status page	Every 30 min (15 min for full outages)
SEV2	Major feature degraded, workaround exists, or significant subset of users	IC + on-call team, status page if customer-visible	Every 60 min
SEV3	Minor degradation, low customer impact	On-call handles in business hours	At milestones
SEV4	No customer impact (internal tooling, near miss)	Ticket	None

When unsure between two levels, pick the higher one; downgrade later.

Roles

Role	Does	Does not
Incident Commander (IC)	Owns decisions, sets priorities, assigns work, declares resolved	Debug hands-on
Ops / Tech lead	Investigates and executes mitigations	Write external comms
Comms lead	Internal updates, status page, support/CS liaison	Change production
Scribe	Timeline, decisions, links, who did what	Filter or editorialize

In small teams one person may hold two roles, but the IC should not be the one typing commands for a SEV1. Claude typically acts as scribe, analyst, and comms drafter.

Workflow

- [ ] 0-5 min:   Declare, assign IC, open channel/doc, set severity
- [ ] 5-15 min:  Assess impact and blast radius; correlate with recent changes
- [ ] 15-30 min: Apply the safest mitigation; first external update
- [ ] Ongoing:   One change at a time, watch metrics, updates on cadence
- [ ] Resolve:   Metrics back to baseline for a monitoring window; declare resolved
- [ ] After:     Final update, hand off to postmortem with timeline and evidence

1. Declare

• Create the incident channel (#inc-2026-10-19-checkout-errors) and an incident doc from the template in references/comms-templates.md.
• Record: detected at, detected by (alert/customer), severity, IC, current impact.

2. Assess: impact and what changed

Answer these first, in order:

1. Who is affected and how? (all users, a region, a tenant, a platform, one endpoint)
2. Since when exactly? Find the inflection point on the error-rate or latency graph.
3. What changed right before it?

# Deploys and config changes near the inflection point
git log --since="3 hours ago" --oneline --merges
gh run list --workflow deploy.yml --limit 10
kubectl rollout history deploy/<svc> -n <ns>
kubectl get events -A --sort-by=.lastTimestamp | tail -40
# Feature flag audit log, infra change log (Terraform runs), cloud provider status page,
# third-party status pages (payments, auth, CDN, DNS)

Change sources to check: application deploys, config/flag changes, infra (Terraform, k8s manifests), dependency or certificate expiry, traffic spikes (marketing, bots), upstream providers, scheduled jobs/cron, data migrations.

For log/metric digging during triage, the log-analysis skill has query patterns.

3. Mitigate

Situation	First mitigation	Rollback of the mitigation
Started after a deploy	Roll back to previous version	Redeploy the new version
Started after a flag/config change	Revert the flag/config	Re-enable
One region/zone/node pool bad	Drain or fail over traffic	Restore routing
Overload / traffic spike	Scale out; rate-limit or shed non-critical traffic	Scale back; remove limits
Bad actor / bot flood	Block at WAF/CDN by IP/ASN/path	Remove rule
Dependency down	Circuit-break, serve degraded/cached, queue writes	Close breaker
Database saturated	Kill runaway queries, pause batch jobs, add read replicas	Resume jobs
Disk full	Expand volume, rotate/compress logs	(Expansion is not reversed)
Cert expired	Renew/redeploy cert	n/a

State each proposed action like this before executing:

PROPOSED (14:22 UTC): Roll back checkout-api from 1.14.2 to 1.14.1
Expected effect: 5xx rate drops from 12% to baseline (<0.5%) within 5 min
Risk: 1.14.2 migration added a nullable column; 1.14.1 ignores it (safe)
Rollback: redeploy 1.14.2
Needs confirmation from IC.

4. Communicate on cadence

• Post the first external update within 30 minutes of declaring a customer-facing SEV1/2, even if it only says “investigating”.
• Every update states: current impact, what is being done, next update time.
• Internal updates can include hypotheses; external ones cannot.
• Use the templates in references/comms-templates.md.

5. Resolve

• Mitigated: impact stopped, root cause may still be present.
• Resolved: key metrics at baseline for a monitoring window (30–60 min for SEV1), backlogs drained (queues, retries), no data repair pending or repair tracked.
• Final update internally and externally. Then hand off to the postmortems skill with the timeline, graphs, and decisions.

Timeline discipline

Log entries as HH:MM UTC — who — what (evidence link):

14:02 UTC — PagerDuty — Alert: checkout 5xx > 5% for 5 min
14:05 UTC — @maria — Declared SEV1, IC @maria, scribe @claude
14:09 UTC — @dev1 — Error spike aligns with deploy of checkout-api 1.14.2 at 13:58
14:22 UTC — @dev1 — Rollback to 1.14.1 started (approved by IC)
14:27 UTC — @dev1 — 5xx back to 0.3%; monitoring
15:05 UTC — @maria — Resolved after 38 min at baseline

Gotchas

• Debugging in silence: if no one has posted in 15 minutes, the IC asks for status.
• Multiple people changing production at once makes it impossible to know what worked.
• Rolling back application code does not roll back migrations, data writes, or cache contents; check those explicitly.
• Restarting everything destroys evidence (heap state, logs on ephemeral disks). Grab a heap dump, thread dump, or logs from one instance before restarting, if it costs under a minute.
• Status page silence is worse than “still investigating”.
• Declaring resolved at the first good graph point; wait for the monitoring window.
• Fatigue: rotate the IC and responders on incidents longer than ~3 hours and do a written handover.

Output

The incident doc (template in references) with: summary, severity, roles, impact, timeline, actions taken with approvals, comms sent, open follow-ups, and links to graphs. At handover or resolution, post a five-line summary in the channel:

**Status:** Resolved 15:05 UTC (SEV1, 63 min, customer impact 25 min)
**Impact:** ~18% of checkout attempts failed with 500 between 13:58 and 14:27 UTC
**Mitigation:** Rolled back checkout-api 1.14.2 -> 1.14.1
**Suspected trigger:** 1.14.2 deploy (root cause TBD in postmortem)
**Follow-ups:** Postmortem Thu; failed orders being retried (ticket OPS-812)