Log Analysis skill for Claude Code

Log analysis is done when you can state what happened, when it started, how many requests or users it affected, and which evidence (queries plus sample lines) supports it, compared against a normal baseline period. A pile of scary-looking errors is not a finding until you know they are new.

Ground rules

• Fix the question and the time window first (UTC). Unbounded searches waste time and money.
• Always compare against a baseline: same window yesterday or before the change. Many “errors” happen all day.
• Count before you read. Group by signature, then read samples from the top groups.
• Follow one request end to end with its request ID or trace ID before generalizing.
• Never paste secrets, tokens, or personal data from logs into reports; redact samples.
• Cloud log queries cost money per GB scanned; narrow by time and log group/stream first.

Workflow

- [ ] 1. Question, time window, affected service(s)
- [ ] 2. Locate the logs and their format
- [ ] 3. Volume over time: find the inflection point
- [ ] 4. Group errors into signatures; diff incident window vs. baseline
- [ ] 5. Trace a few failing requests end to end
- [ ] 6. Correlate with deploys, config changes, dependencies
- [ ] 7. Write findings with queries and redacted samples

2. Locate

Where	Command
Local files	ls -lh /var/log/app/; rotated: *.gz -> zcat, zgrep
systemd	journalctl -u nginx --since "2026-10-19 13:30" --until "2026-10-19 14:30" -o json
Docker	docker logs --since 1h --timestamps <ctr>
Kubernetes	kubectl logs -l app=api -n shop --since=1h --timestamps --prefix --max-log-requests=20
Crashed pod	kubectl logs <pod> --previous
Loki / CloudWatch / others	See references/query-cheatsheet.md

Check the format on three lines before writing any pipeline: JSON per line, logfmt, or free text (nginx combined, syslog).

3. Volume over time

# JSON logs: errors per minute
jq -r 'select(.level=="error") | .timestamp[0:16]' app.log | sort | uniq -c

# nginx combined log: 5xx per minute
awk '$9 ~ /^5/ {print substr($4, 2, 17)}' access.log | sort | uniq -c

# Status code distribution
awk '{print $9}' access.log | sort | uniq -c | sort -rn

The first minute where the count departs from baseline is the start time. Note it.

4. Signatures and baseline diff

Normalize variable parts (IDs, numbers, UUIDs, IPs) so identical errors group together:

normalize() {
  sed -E 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<uuid>/g;
          s/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/<ip>/g;
          s/0x[0-9a-f]+/<hex>/g;
          s/[0-9]+/<n>/g'
}
jq -r 'select(.level=="error") | .msg' incident.log  | normalize | sort | uniq -c | sort -rn > incident.sig
jq -r 'select(.level=="error") | .msg' baseline.log  | normalize | sort | uniq -c | sort -rn > baseline.sig
head -20 incident.sig
# Signatures present now but absent (or rare) in baseline:
awk 'NR==FNR {b[substr($0, index($0,$2))]=$1; next}
     {sig=substr($0, index($0,$2)); if (!(sig in b) || $1 > 5*b[sig]) print}' baseline.sig incident.sig

Other high-value groupings:

jq -r 'select(.status>=500) | .path' app.log | sed -E 's#/[0-9]+#/:id#g' | sort | uniq -c | sort -rn | head
jq -r 'select(.level=="error") | "\(.service) \(.error.type // "none")"' app.log | sort | uniq -c | sort -rn
# Slowest requests
jq -c 'select(.duration_ms > 1000) | {t:.timestamp, path, duration_ms, request_id}' app.log | head
# p50/p95/p99 duration from JSON logs
jq -r '.duration_ms // empty' app.log | sort -n | awk '{a[NR]=$1} END {print "p50",a[int(NR*.5)],"p95",a[int(NR*.95)],"p99",a[int(NR*.99)]}'

5. Follow one request

RID=7f3c2a91
rg -z "$RID" /var/log/app/ /var/log/nginx/                 # every line, every service
jq -c --arg rid "$RID" 'select(.request_id==$rid)' *.log | jq -s 'sort_by(.timestamp)[] | {timestamp, service, level, msg}'

If services share a trace_id, use it to jump into the trace view (OpenTelemetry) for timing; logs explain what, traces explain where time went.

6. Correlate

Line up the inflection point with deploy times, config/flag changes, cron schedules, traffic changes, and dependency status pages. A correlation is a lead, not proof; look for an error signature that directly names the changed component.

Tool choice

Need	Tool
JSON lines	jq (-c compact, -r raw, select, group_by after -s)
Fast search across many/big files	rg (ripgrep, handles .gz with -z), or LC_ALL=C grep -F
Column-based text logs	awk
Live tail with filtering	`tail -F file
Many GB, repeated questions	Load into DuckDB: SELECT ... FROM read_json_auto('app.log')
Centralized logs	LogQL / Logs Insights / KQL — see reference

Structured logging standard

When asked to improve logs, emit one JSON object per line with these fields:

{"timestamp":"2026-10-19T14:02:11.483Z","level":"error","service":"checkout",
 "env":"prod","version":"1.14.2","msg":"payment authorization failed",
 "request_id":"7f3c2a91","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
 "http":{"method":"POST","path":"/orders","status":502},"duration_ms":2310,
 "error":{"type":"UpstreamTimeout","message":"stripe timeout after 2000ms"}}

• msg is a constant string; variable data goes in fields (so grouping works).
• Timestamps in UTC ISO 8601 with milliseconds.
• Generate a request ID at the edge, propagate it in a header, include it in every line.
• Levels: error = needs attention, warn = unexpected but handled, info = business events, debug off in prod. One error log per failure, at the boundary that handles it.
• Never log secrets, tokens, full card numbers, passwords; hash or drop user identifiers according to policy.

Gotchas

• Mixed time zones between services make correlation lie; convert everything to UTC.
• Log sampling or rate limiting at the agent drops lines during exactly the spikes you care about; check for “dropped” counters.
• Multi-line stack traces split into separate lines in naive shippers; grep finds the first line only.
• kubectl logs returns only the current container’s log and only what is on the node; rotated or evicted pod logs need the central store.
• grep -c error matches “error_count=0”; anchor on the level field.
• Errors that also occur in baseline at the same rate are background noise, not the cause.

Output

**Question:** Why did checkout fail between 13:58 and 14:27 UTC?
**Start:** 13:58:40 UTC (errors/min 3 -> 410); baseline same window yesterday: 2–5/min
**Top new signature (92% of errors):** "payment authorization failed" error.type=UpstreamTimeout
  - 11,842 occurrences, all from checkout 1.14.2 pods, none from 1.14.1
**Example request:** request_id 7f3c2a91 — POST /orders -> payments call 2.0 s timeout -> 502
**Correlation:** checkout 1.14.2 deployed 13:58:05; changed payments client timeout 10s -> 2s
**Queries used:** (listed, with time ranges)