Runbooks skill for Claude Code

A runbook is done when an on-call engineer who has never seen this alert can go from page to mitigation using only the runbook and their standard access, every command runs as written, risky steps are marked and gated, and the alert links straight to it.

Ground rules

• One runbook per alert (or per tightly related alert group). The alert’s runbook_url annotation points to it.
• Assume a tired reader with no context: no “as usual”, no “check the dashboard” without a link, no unexplained acronyms.
• Mitigate first, diagnose fully later. Order actions by safety and speed: read-only checks → reversible mitigations → risky actions.
• Commands are copy-pasteable with explicit placeholders and expected output.
• Destructive or customer-visible steps (restarts of stateful systems, failovers, data deletion, scaling down) require explicit confirmation from the incident commander or a second engineer, stated inline.
• Every runbook has an owner and a last-tested date; untested runbooks rot.

Workflow

- [ ] 1. Start from the alert definition: query, threshold, duration, severity
- [ ] 2. Write what it means and customer impact
- [ ] 3. Write first-5-minutes triage (read-only)
- [ ] 4. Build the diagnosis decision tree from past incidents
- [ ] 5. Write mitigations with commands, risk level, and verification
- [ ] 6. Add escalation and communication steps
- [ ] 7. Test in staging or a game day; link from the alert

1. Link from the alert

# Prometheus alerting rule
- alert: CheckoutHighErrorRate
  expr: |
    sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{service="checkout"}[5m])) > 0.02
  for: 5m
  labels: { severity: page, team: payments }
  annotations:
    summary: "Checkout 5xx rate {{ $value | humanizePercentage }} over 5m"
    runbook_url: "https://runbooks.example.com/checkout/high-error-rate"
    dashboard: "https://grafana.example.com/d/checkout?from=now-1h"

3. First five minutes (read-only)

Write the exact commands and what “normal” looks like:

# 1. Is it all pods or some? (normal: all Running, restarts stable)
kubectl -n checkout get pods -o wide

# 2. Recent deploys in the last 2 hours (most incidents follow a change)
kubectl -n checkout rollout history deploy/checkout-api | tail -5

# 3. Error breakdown by route (normal: < 0.5% on every route)
#    Grafana: Checkout → "5xx by route" panel (link above)

# 4. Dependency health (normal: p95 < 50 ms)
kubectl -n checkout logs deploy/checkout-api --since=10m | grep -E 'timeout|ECONNREFUSED' | tail -20

4. Diagnosis decision tree

- Errors started within 30 min of a deploy?
  → Yes: go to Mitigation A (roll back).
- Errors only on routes that call the payments provider?
  → Yes: check provider status page (link); go to Mitigation B (enable fallback).
- Pods restarting with OOMKilled?
  → Yes: go to Mitigation C (raise memory limit temporarily).
- Database connection errors in logs?
  → Yes: go to "DB connection saturation" runbook (link).
- None of the above → escalate (see Escalation) and keep collecting: error samples,
  traces for 3 failing request IDs.

5. Mitigations

Each mitigation: when to use, risk, command, expected effect, verification, how to undo.

#### Mitigation A: Roll back the last deploy  (risk: low, reversible)
When: errors began after a deploy.

kubectl -n checkout rollout undo deploy/checkout-api
kubectl -n checkout rollout status deploy/checkout-api --timeout=5m

Expected: rollout completes in ~3 min; 5xx rate under 0.5% within 5 min on the dashboard.
Undo: redeploy the reverted version from the CD pipeline once fixed.
Then: post in #inc channel; block the bad version in CD.

#### Mitigation D: Fail over the primary database  (risk: HIGH, customer-visible)
Requires: explicit go-ahead from the incident commander, stated in the incident channel.
Expect 30–60 s of write errors during failover. ...

Risk labels: low (read-only or instantly reversible), medium (reversible, brief impact), high (customer-visible or hard to reverse; requires IC approval).

6. Escalation and communication

## Escalation
- 15 min without mitigation → page secondary on-call (payments-secondary in PagerDuty).
- Payments provider issue → open ticket with provider (support portal link, account ID
  in 1Password "Payments vendor").
- Data integrity concern → page database on-call and notify #security if PII may be
  affected.

## Communication
- SEV2+: incident commander posts status page update within 15 min (template link).
- Customer support: post in #support-incidents with impact summary.

Writing rules (before / after)

Before	After
“Check if the DB is OK.”	“Run psql "$CHECKOUT_DB_URL" -c "select count(*) from pg_stat_activity where state='active';" — normal is < 80; above 180 means pool saturation (go to Mitigation E).”
“Restart the service.”	“Rolling restart (low risk): kubectl -n checkout rollout restart deploy/checkout-api. Expect no errors; pods replace one at a time.”
“Look at the logs.”	“kubectl -n checkout logs deploy/checkout-api --since=15m | grep -c 'payment_timeout' — more than 50 means provider latency; go to Mitigation B.”
“Escalate if needed.”	“If not mitigated in 15 minutes, page payments-secondary.”

• Use placeholders that cannot run by accident: <ORDER_ID>, and state where to get them.
• Keep secrets out; reference the vault entry by name.
• Prefer links to dashboards with a preset time range over “open Grafana”.

Gotchas

• Runbooks that require access on-call does not have (prod DB write, cloud console roles); verify with an on-call engineer’s account.
• Copy-paste hazards: smart quotes, line wrapping, commands that default to the wrong context (kubectl current-context, aws profile). Include --context/-n/--profile explicitly.
• Stale runbooks after a migration point to dead hosts or dashboards; add link checking and a review date.
• Long prose before the first command; the first actionable step should be visible without scrolling.
• Automatable steps that stay manual forever; if a runbook step is run every week, file a ticket to automate it.

Verification

• A teammate unfamiliar with the service follows the runbook in staging (or a game day) and reaches mitigation without asking questions; fix every point of confusion.
• Every command runs in the documented context and shows the documented normal output.
• runbook_url is set on the alert and resolves; links checked (lychee runbooks/).
• Owner and “last tested” date are present and less than 6 months old.

Output contract

# <Alert name>
Owner: <team>   Last tested: <YYYY-MM-DD>   Severity: <page | ticket>
Alert source: <link to rule>   Dashboard: <link with time range>

## What this alert means
<1–2 sentences + threshold>

## Customer impact
## First 5 minutes (read-only)
## Diagnosis
## Mitigations (A, B, C … with risk, commands, expected result, undo)
## Verify recovery
## Escalation
## Communication
## Follow-up (postmortem threshold, tickets)
## History (past incidents using this runbook)