Linux Troubleshooting skill for Claude Code
Name the saturated resource, with evidence. Claude runs a read-only 60-second checklist before touching anything, then follows the evidence to the saturated resource: run queue, D-state tasks, swapping, disk await, socket backlogs or fd limits. A bundled vmstat analyzer ranks suspects from captured output, and every finding ends with the responsible process, the cause and a mitigation that waits for your approval.
- name
linux-troubleshooting - size174 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Diagnoses slow or failing Linux hosts with a USE-method triage - load and CPU saturation, memory pressure, OOM kills and swap, disk latency and full filesystems or inodes, network sockets, retransmits and DNS, file descriptor limits, and stuck processes - using uptime, vmstat, mpstat, pidstat, iostat, free, df, ss, dmesg, journalctl, strace, lsof and /proc, plus a bundled vmstat analyzer. Use when a server is slow, load is high, a process was killed, a disk is full, connections time out, or the user pastes top, vmstat or iostat output.Try asking
- Load average on this box is 40 but CPU looks idle. What is going on?
- Here is our vmstat and iostat output from the slowdown. Find the bottleneck
- The disk says full but du does not add up. Track down where the space went
Use it when
- A Linux server is slow, overloaded or unresponsive
- A process was OOM-killed or hit too many open files
- A filesystem is full or out of inodes
- Connections time out or get refused from a host
- You have top, vmstat, iostat or ss output to interpret
Skip it when
- You are searching application logs for an error pattern — use log-analysis
- The slowness is inside one Python or Node process — use python-profiling or node-profiling
How to check the result
The report names one saturated resource backed by at least two tools' output, identifies the responsible process, and the metric returns to baseline after the approved mitigation.
The common mistake it prevents
Deleting a huge log file that a running process still holds open, which frees no space until the writer is restarted.
--- name: linux-troubleshooting description: Diagnoses slow or failing Linux hosts with a USE-method triage - load and CPU saturation, memory pressure, OOM kills and swap, disk latency… license: MIT ---
Triage is done when you can name the saturated resource (CPU, memory, disk, network, or a limit), show the evidence from at least two tools, identify the process or cause responsible, and state a mitigation with its risk. “Load is high” is a symptom, not a diagnosis.
Ground rules
- Observe before acting. Capture the 60-second checklist output first; the evidence disappears once you restart things.
- Read-only commands first. Killing processes, dropping caches, restarting services, deleting files or rebooting require explicit user confirmation, especially in production.
- Never
rma large log that a process holds open; the space is not freed. Truncate (: > file) or restart the writer, after confirming. strace/perfadd overhead; attach briefly (-p PIDfor seconds) on production.- Prefer facts from the kernel (
/proc,dmesg, PSI) over intuition fromtop.
The 60-second checklist
Run in order and save the output (sysstat package provides mpstat/pidstat/iostat/sar):
uptime # load averages: 1/5/15 min trend
dmesg -T | tail -50 # OOM kills, disk errors, NIC resets, segfaults
journalctl -p warning --since "1 hour ago" --no-pager | tail -50
vmstat 1 10 # r, b, si/so, wa, st
mpstat -P ALL 1 3 # per-CPU imbalance, %iowait, %steal
pidstat 1 3 # which process uses CPU
iostat -xz 1 3 # per-device util, await, aqu-sz
free -m # available memory, swap use
df -hT; df -i # space and inodes
ss -s; ss -tanp state established | head # socket summary, top connections
cat /proc/pressure/{cpu,memory,io} # PSI stall percentages (kernel 4.20+)Analyze captured vmstat output with the bundled script:
vmstat 1 30 > /tmp/vmstat.txt
python3 scripts/vmstat_triage.py /tmp/vmstat.txt --cpus "$(nproc)"It flags run-queue saturation (r > CPUs), blocked tasks (b), swapping (si/so), high iowait, steal, and context-switch storms, prints the dominant suspect, and exits 1 when any resource is saturated.
Decision table
| Evidence | Points to | Next command |
|---|---|---|
r > CPU count, us+sy near 100 | CPU saturation | pidstat -u 1, top -H -p PID, perf top -p PID |
High load but low CPU, b > 0, high wa | Disk I/O wait (or NFS) | iostat -xz 1, pidstat -d 1, iotop -o |
si/so non-zero continuously, low available | Memory pressure / swapping | `ps aux —sort=-rss |
st > 5-10% | Noisy neighbor / CPU credit exhaustion (cloud) | Check instance type, burst credits, move host |
sy high, huge cs/in | Lock contention, syscall storm, too many threads | pidstat -w 1, perf top, strace -c -p PID |
| One CPU at 100%, others idle | Single-threaded bottleneck or IRQ affinity | mpstat -P ALL 1, cat /proc/interrupts |
| “Killed” / exit 137 | OOM killer | `dmesg -T |
| Writes fail “No space left on device” | Full disk or full inodes | df -h, df -i, `du -xh —max-depth=1 / |
| Connections time out / refused | Backlog, conntrack, port exhaustion, firewall, DNS | ss -ltn (Recv-Q vs Send-Q), ss -s, nstat, dig |
| “Too many open files” | fd limit | `ls /proc/PID/fd |
| Process stuck, not using CPU | Blocked in syscall (D state) or waiting on lock/network | cat /proc/PID/stack, cat /proc/PID/wchan, strace -p PID |
Deep dives
CPU
top -o %CPU # press 1 for per-CPU, H for threads
ps -eo pid,ppid,stat,pcpu,pmem,etime,cmd --sort=-pcpu | head -15
pidstat -t -p <PID> 1 5 # per-thread CPU
perf top -p <PID> # hot functions (needs perf, symbols)
perf record -F 99 -g -p <PID> -- sleep 15 && perf report # then flame graph if neededLoad average counts runnable plus uninterruptible (D) tasks on Linux. High load with idle CPU means D-state tasks: usually disk or NFS.
Memory
free -m # use "available", not "free"
grep -E "MemAvailable|Dirty|Writeback|Slab|SReclaimable|Committed_AS" /proc/meminfo
ps aux --sort=-rss | head -10
cat /sys/fs/cgroup/<path>/memory.events # cgroup v2: oom, oom_kill, high counts
slabtop -o | head -15 # kernel slab growth (dentry/inode caches)- Page cache is not a leak;
availableaccounts for reclaimable cache. - Containers get OOM-killed at their cgroup limit while the host shows free memory.
- RSS steadily rising over days in one process: application leak (see memory-leak-hunting).
Disk
iostat -xz 1 5 # r_await/w_await ms, aqu-sz, %util
pidstat -d 1 5 # per-process read/write kB/s
du -xh --max-depth=1 /var | sort -h | tail
lsof +L1 # deleted files still held open (space not freed)
find / -xdev -type f -size +1G -printf '%s %p\n' 2>/dev/null | sort -n | tail| Metric | Healthy (SSD/NVMe) | Problem |
|---|---|---|
| await | < 1-5 ms | > 20 ms sustained |
| aqu-sz | < 1 per device | Growing queue |
| %util | Not reliable for NVMe/RAID (parallel devices) | Use await and queue instead |
Inodes full (df -i 100%) with space left: millions of small files (sessions, cache,
mail queue). Find with find /path -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; | sort -n | tail.
Network
ss -ltnp # listening sockets; Recv-Q > 0 on LISTEN = accept backlog full
ss -tan state time-wait | wc -l # TIME_WAIT pile-up
ss -tin dst 10.0.2.15 # per-connection rtt, retrans, cwnd
nstat -az | grep -E "TcpRetransSegs|ListenOverflows|ListenDrops|TcpExtTCPTimeouts"
ip -s link # interface errors/drops
conntrack -S 2>/dev/null; sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
dig +short api.example.com; resolvectl status # DNS path
curl -w '@-' -o /dev/null -s https://api.example.com <<< 'dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer}\n'ListenOverflowsincreasing: app accepts too slowly or backlog too small (net.core.somaxconn, app listen backlog).- Ephemeral port exhaustion on clients making many outbound connections: reuse
connections (keepalive pools) before tuning
ip_local_port_range. nf_conntrack: table fullin dmesg drops packets silently.
Processes and limits
cat /proc/<PID>/status | grep -E "State|Threads|VmRSS"
cat /proc/<PID>/limits
ls /proc/<PID>/fd | wc -l
lsof -p <PID> | awk '{print $5}' | sort | uniq -c | sort -rn | head # fd types
strace -f -tt -T -p <PID> -o /tmp/trace.txt # Ctrl-C after a few seconds
strace -c -p <PID> # syscall summary
systemctl show <unit> -p LimitNOFILE # systemd overrides ulimitGotchas
top%CPU is per core (400% = 4 cores busy).vmstatpercentages are of all CPUs.- The first line of
vmstatandiostatis the average since boot; ignore it. ulimit -nin your shell is not the service’s limit; check/proc/PID/limits.- Containers hide host pressure: run the checklist on the node, and read cgroup files
for the container (
memory.current,memory.max,cpu.statthrottled_usec). - CPU throttling under a cgroup CPU limit looks like latency, not high CPU. Check
nr_throttledincpu.stat. - Time skew breaks TLS, auth tokens and distributed locks:
timedatectlandchronyc tracking. echo 3 > /proc/sys/vm/drop_cachesfixes nothing and causes an I/O storm.
Report
**Host:** api-3 (8 vCPU, 16 GiB) **Window:** 14:05-14:20 UTC
**Saturated resource:** disk I/O on nvme1n1 (w_await 180 ms, aqu-sz 34)
**Evidence:** vmstat b=6-11, wa=48%; pidstat -d shows postgres checkpointer 220 MB/s;
dmesg clean; CPU us 12%
**Cause:** max_wal_size too small -> checkpoints every 40 s during bulk import
**Mitigation (proposed, needs approval):** pause import job; raise max_wal_size to 8GB
**Follow-up:** alert on disk await p95 > 50 ms for 10 minBundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/vmstat_triage.py
#!/usr/bin/env python3
"""Summarize `vmstat 1 N` output and flag saturated resources.
Usage:
vmstat 1 30 > vmstat.txt
python3 scripts/vmstat_triage.py vmstat.txt --cpus 8
vmstat 1 30 | python3 scripts/vmstat_triage.py - --cpus "$(nproc)"
Skips the first sample (averages since boot). Exit 1 if any resource is saturated.
"""
import argparse
import statistics
import sys
def parse(lines):
header = None
rows = []
for line in lines:
parts = line.split()
if not parts:
continue
if "r" in parts and "b" in parts and "us" in parts:
header = parts
continue
if header and all(p.lstrip("-").isdigit() for p in parts) and len(parts) >= len(header):
rows.append(dict(zip(header, map(int, parts[: len(header)]))))
return rows
def main():
ap = argparse.ArgumentParser()
ap.add_argument("file", help="vmstat output file or - for stdin")
ap.add_argument("--cpus", type=int, required=True, help="number of CPUs (nproc)")
ap.add_argument("--keep-first", action="store_true", help="do not drop the since-boot sample")
a = ap.parse_args()
text = sys.stdin.read() if a.file == "-" else open(a.file, encoding="utf-8").read()
rows = parse(text.splitlines())
if not a.keep_first and len(rows) > 1:
rows = rows[1:]
if not rows:
print("no vmstat samples found")
return 2
def col(name):
return [r.get(name, 0) for r in rows]
def summary(name):
v = col(name)
return statistics.mean(v), max(v)
n = len(rows)
findings = []
r_mean, r_max = summary("r")
b_mean, b_max = summary("b")
si_mean, _ = summary("si")
so_mean, _ = summary("so")
wa_mean, wa_max = summary("wa")
st_mean, _ = summary("st")
us_mean, _ = summary("us")
sy_mean, _ = summary("sy")
id_mean, _ = summary("id")
cs_mean, _ = summary("cs")
swapping = sum(1 for r in rows if r.get("si", 0) + r.get("so", 0) > 0)
if r_mean > a.cpus:
findings.append(("CPU", f"run queue avg {r_mean:.1f} > {a.cpus} CPUs (max {r_max})", r_mean / a.cpus))
if us_mean + sy_mean > 90:
findings.append(("CPU", f"CPU busy {us_mean + sy_mean:.0f}% (us {us_mean:.0f} sy {sy_mean:.0f})", 1.5))
if b_mean >= 1:
findings.append(("DISK", f"blocked (D-state) tasks avg {b_mean:.1f} (max {b_max})", 1 + b_mean / 2))
if wa_mean > 20:
findings.append(("DISK", f"iowait avg {wa_mean:.0f}% (max {wa_max})", 1 + wa_mean / 40))
if swapping > n / 3:
findings.append(("MEMORY", f"swapping in {swapping}/{n} samples (si {si_mean:.0f} so {so_mean:.0f} per s)", 2 + so_mean / 100))
if st_mean > 10:
findings.append(("CPU-STEAL", f"steal avg {st_mean:.0f}%: hypervisor contention or burst credits exhausted", 1 + st_mean / 20))
if sy_mean > 30 and sy_mean > us_mean:
findings.append(("KERNEL", f"system time {sy_mean:.0f}% exceeds user time: syscalls, locks or interrupts", 1.2))
if cs_mean > 50000 * a.cpus / 8:
findings.append(("KERNEL", f"context switches avg {cs_mean:.0f}/s: thread contention likely", 1.1))
print(f"samples: {n} cpus: {a.cpus}")
print(f"r {r_mean:.1f}/{r_max} b {b_mean:.1f}/{b_max} us {us_mean:.0f} sy {sy_mean:.0f} "
f"id {id_mean:.0f} wa {wa_mean:.0f} st {st_mean:.0f} si {si_mean:.0f} so {so_mean:.0f} cs {cs_mean:.0f}")
if not findings:
print("no saturation detected in this window; check application latency, locks, or downstream services")
return 0
findings.sort(key=lambda f: f[2], reverse=True)
for res, msg, _ in findings:
print(f"SATURATED {res:10} {msg}")
print(f"dominant suspect: {findings[0][0]}")
return 1
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.