Go pprof Profiling skill for Claude Code

An optimization is real when a profile of a realistic workload pointed at it, and benchstat (or production metrics) shows a statistically significant improvement afterwards. Guessing at hot paths in Go is usually wrong — allocation and GC cost hide in places that look cheap.

Pick the profile for the symptom

Symptom	Profile	Notes
High CPU, slow handlers, throughput ceiling	profile (CPU)	100 Hz sampling; collect ≥ 30 s under load
RSS grows, OOM kills	heap, -sample_index=inuse_space	What is live now (as of last GC)
High GC CPU (runtime.gcBgMarkWorker, mallocgc in CPU profile)	heap / allocs, -sample_index=alloc_space or alloc_objects	What churns, even if freed quickly
Goroutine count climbs	goroutine (?debug=1 groups by stack)	Compare two snapshots minutes apart
Latency high but CPU low	block, mutex, then trace	Waiting on channels, locks, I/O, scheduler
Tail latency spikes, GC pauses, scheduler delays	execution trace (go tool trace)	Timeline per P/goroutine; flight recorder for rare events
Leaked goroutines blocked forever (Go 1.26)	goroutineleak	Experimental: build with GOEXPERIMENT=goroutineleakprofile

Workflow

- [ ] 1. Reproduce with a realistic load (benchmark or load test), record baseline numbers
- [ ] 2. Collect the right profile for the symptom
- [ ] 3. Read it: top -cum, flame graph, list on the hottest frames
- [ ] 4. Form one hypothesis; change one thing
- [ ] 5. Re-profile and benchstat old vs new (count ≥ 10)
- [ ] 6. Confirm in production metrics or a production profile

Collecting

From benchmarks (most reproducible)

func BenchmarkEncode(b *testing.B) {
    in := loadFixture(b)          // setup outside the loop
    b.ReportAllocs()
    for b.Loop() {                 // Go 1.24+: keeps results alive, excludes setup
        _ = Encode(in)
    }
}

go test -run='^$' -bench=BenchmarkEncode -benchmem -count=10 \
  -cpuprofile cpu.out -memprofile mem.out ./internal/codec | tee old.txt
# also available: -blockprofile block.out -mutexprofile mutex.out -trace trace.out

Before Go 1.24 use for i := 0; i < b.N; i++ and assign results to a package-level sink so the compiler cannot delete the work.

From a running service

import (
    "net/http"
    "net/http/pprof"
)

func startDebugServer() {
    mux := http.NewServeMux()
    mux.HandleFunc("/debug/pprof/", pprof.Index)
    mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
    mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
    go http.ListenAndServe("127.0.0.1:6060", mux) // never on the public listener
}

The blank import _ "net/http/pprof" registers handlers on http.DefaultServeMux; if the public server uses the default mux, profiles (including command lines and heap contents) become public. Use an explicit mux on a loopback or internal port, reached via kubectl port-forward or SSH tunnel.

go tool pprof -http=:8080 'http://localhost:6060/debug/pprof/profile?seconds=30'
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=1' | head -50
curl -s -o trace.out 'http://localhost:6060/debug/pprof/trace?seconds=5' && go tool trace trace.out

Block and mutex profiles are empty until enabled:

runtime.SetMutexProfileFraction(100)  // sample ~1 in 100 contention events
runtime.SetBlockProfileRate(10_000)   // sample blocking events, ~1 per 10µs blocked

Rate 1 records everything and is too expensive to leave on in production.

Programmatically (CLIs, batch jobs)

f, _ := os.Create("cpu.out")
pprof.StartCPUProfile(f)          // runtime/pprof
defer pprof.StopCPUProfile()
// ... at the end, for memory:
runtime.GC()                      // so inuse reflects live data
pprof.Lookup("heap").WriteTo(memFile, 0)

Reading profiles

go tool pprof cpu.out
(pprof) top -cum 20          # who is on the stack most (inclusive)
(pprof) top 20               # where CPU is actually burned (flat)
(pprof) list Encode          # line-level cost inside a function
(pprof) peek 'json\.'        # callers and callees of matching functions
(pprof) web                  # call graph (needs graphviz); -http UI has flame graph

• flat = time in the function itself; cum = including callees. Start from high cum in your own packages, then follow down to high flat.
• In the web UI (-http), the flame graph is the default view in recent Go releases; use “Refine → Focus/Ignore” or flags -focus=pkg/codec -ignore=runtime.
• Heap profiles have four sample indexes: inuse_space, inuse_objects, alloc_space, alloc_objects. Leaks → inuse; GC pressure → alloc. Pass -sample_index=alloc_space.
• Compare two profiles: go tool pprof -http=:8080 -diff_base=old.cpu new.cpu (negative = improvement). For goroutine or heap growth: -base=heap1.out heap2.out.

Signatures in a CPU profile and what they mean:

Hot frames	Meaning	Next step
runtime.mallocgc, gcBgMarkWorker, scanobject	Allocation and GC pressure	alloc_space profile; cut allocations
runtime.mapaccess*, mapassign*	Map-heavy hot loop	Pre-size maps, use slices/structs for small fixed keys
runtime.memmove, growslice	Slices growing repeatedly	make([]T, 0, n)
encoding/json.*, reflect.*	Reflection-based encoding	Generated codecs, fewer round trips, cache encoders
syscall.Syscall, runtime.netpoll	I/O bound	Buffering (bufio), batching, fewer small writes
runtime.futex, lock2, chanrecv, selectgo	Contention or waiting	Mutex/block profile, trace
regexp.* with Compile	Regex compiled per call	Compile once at package level

Common fixes, in order of payoff

1. Remove work: cache results, avoid repeated parsing/encoding, hoist invariants out of loops.
2. Cut allocations: preallocate slices and maps, strings.Builder with Grow, reuse buffers, avoid []byte↔string round trips, avoid fmt.Sprintf in hot paths.
3. Check escape analysis: go build -gcflags='-m=2' ./pkg 2>&1 | grep -E 'escapes|moved to heap'. Returning pointers to locals, storing in interfaces, and closures capturing variables move values to the heap.
4. sync.Pool only for large, frequently reused buffers, and only with a profile showing their allocation; reset before Put, never pool objects that hold references you still use.
5. Reduce contention: shard locks, sync.RWMutex only when reads dominate heavily, atomic counters, per-goroutine accumulation then merge.
6. Profile-guided optimization: save a representative production CPU profile as default.pgo in the main package directory; go build uses it automatically (Go 1.21+). Typical gain is a few percent of CPU; refresh it periodically.

Runtime knobs (after code fixes, not instead)

• GOMEMLIMIT (e.g. 90% of the container memory limit) lets the GC work harder near the limit instead of OOMing; pair with default GOGC=100, or GOGC=off + GOMEMLIMIT for memory-bounded batch jobs only.
• GOMAXPROCS: Go 1.25+ respects cgroup CPU limits automatically; on older versions in containers use go.uber.org/automaxprocs or set it explicitly, or the scheduler thrashes against the CPU quota.
• Go 1.26 makes the Green Tea garbage collector the default (opt out at build time with GOEXPERIMENT=nogreenteagc, expected to be removed later). Re-baseline GC numbers after upgrading.
• GODEBUG=gctrace=1 prints one line per GC cycle: heap sizes, pause, CPU share.

Gotchas

• Profiling an idle service or a toy input: the profile shows startup and runtime noise. Drive realistic load (load-testing) while collecting.
• A 5-second CPU profile has ~500 samples per core; percentages under ~2% are noise.
• inuse_space reflects the heap at the most recent GC; force runtime.GC() before writing a heap profile in tests, or compare snapshots taken under the same load.
• The heap profile is sampled (runtime.MemProfileRate, default one sample per 512 KiB); tiny allocations are estimated, not counted.
• Benchmarks on a laptop with thermal throttling or other load produce false deltas; close other work, use -count=10, and trust only benchstat-significant results.
• Comparing profiles across Go versions or build flags mixes in compiler changes.
• Goroutine “leaks” that are really unbounded spawning: count by stack with ?debug=1 and look for one stack whose count tracks request volume. In tests, add go.uber.org/goleak (defer goleak.VerifyNone(t)).
• Memory growing while heap profile is flat: look at goroutine stacks, cgo, mmap, or the runtime returning memory lazily (compare /memory/classes/... in runtime/metrics).
• For browser-side or Node memory leaks use memory-leak-hunting / node-profiling.

Verification

go install golang.org/x/perf/cmd/benchstat@latest
go test -run='^$' -bench=BenchmarkEncode -benchmem -count=10 ./internal/codec > new.txt
benchstat old.txt new.txt

Accept the change only when benchstat reports a significant delta (p < 0.05) for sec/op, B/op, or allocs/op in the intended direction and no other benchmark regressed. Then confirm with a production CPU profile diff (-diff_base) or latency/CPU dashboards.

Output

**Symptom:** /export p99 1.8 s, CPU 85% at 300 rps
**Profile:** 30 s CPU under load — 41% cum in `report.Render`, 22% flat in `runtime.mallocgc`
**Cause:** `fmt.Sprintf` per cell and slice growth in `appendRow` (alloc_space: 2.1 GB/30 s)
**Fix:** `strings.Builder` with Grow, preallocated rows (internal/report/render.go:88)
**benchstat:** sec/op −38% (p=0.000 n=10), allocs/op 1840 → 212
**Prod:** CPU 85% → 54% at same rps; p99 1.8 s → 0.9 s