Benchmark Testing skill for Claude Code

Done well means: each benchmark answers one stated question, measures only the code in question with realistic inputs, runs enough samples on a quiet machine to know its own noise, and compares A against B with a statistical test. A claim of “X% faster” comes with sample count, spread, and p-value or confidence interval. One run per side is an anecdote.

Ground rules

• Benchmarks answer “how long does this code take per operation”. “How many requests per second can the service handle” is a load test (load-testing); “where does the time go” is profiling (go-pprof, node-profiling, python-profiling). Profile first if you do not know what to benchmark.
• Use the ecosystem’s harness. Never time a hand-written loop with Date.now() or time.time(): no warmup, no statistics, timer resolution errors, and dead-code risk.
• Compare on the same machine, same build flags, same input, interleaved or back to back. Numbers from different machines or days are not comparable.
• At least 10 samples per side for comparisons (benchstat’s guidance is 10, ideally 20), decided before running. Rerunning until the result looks significant is p-hacking.
• Release/optimized builds only: -O/--release, no debugger, no coverage instrumentation, no NODE_ENV=development.
• Report medians with spread, not a single mean; timing distributions are right-skewed.

Workflow

- [ ] 1. Write the question: which operation, which inputs, what decision depends on it
- [ ] 2. Pick the harness for the language and the level (micro vs macro)
- [ ] 3. Write the benchmark with realistic inputs and dead-code guards
- [ ] 4. Quiet the machine; run an A/A comparison to measure noise
- [ ] 5. Run baseline and candidate with the same sample count
- [ ] 6. Compare statistically; check the change exceeds the noise
- [ ] 7. Confirm with a profile or a macro benchmark if the claim matters
- [ ] 8. Report, or wire into CI with a threshold above measured noise

2. Harness per language

Language	Micro harness	Compare with
Java / JVM	JMH (jmh-java-benchmark-archetype), run as java -jar target/benchmarks.jar, never from the IDE	JMH JSON output (-rf json) plus compare_bench.py, or JMH’s error bars
Go	testing.B with for b.Loop() (Go 1.24+)	benchstat old.txt new.txt
Python	pytest-benchmark in test suites; pyperf for rigorous standalone runs	--benchmark-compare, pyperf compare_to
Rust	Criterion 0.8 or divan; Gungraun (formerly iai-callgrind) for instruction counts	Criterion baselines (--save-baseline, --baseline)
JS / TS	Vitest bench (fixture API in Vitest 5; top-level bench in Vitest 4 and earlier) or tinybench	Vitest bench.compare, stored results via writeResult / bench.from
.NET	BenchmarkDotNet, Release build	Its statistical summary
CLI / whole program (macro)	hyperfine --warmup 3 --runs 20 'old cmd' 'new cmd'	hyperfine’s relative summary, --export-json

Per-tool code is in the sections below; pick the one in use.

3. Writing the benchmark

• Inputs: realistic size and shape (production-like payloads, sizes that cross cache boundaries). Parameterize sizes (args/@Param/sub-benchmarks) instead of guessing one.
• Setup outside the timed region: build inputs before the loop or in a setup fixture.
• Dead-code elimination: the compiler or JIT removes work whose result is unused and folds constant inputs. Consume every result and make inputs opaque:

Harness	Guard
JMH	Return the result from the @Benchmark method or pass it to Blackhole.consume; keep inputs in @State fields, not constants
Go 1.24+	b.Loop() keeps call arguments and results alive; with the old b.N loop, assign to a package-level sink variable
Rust	std::hint::black_box(input) and return the output (criterion::black_box is deprecated)
JS	Use the result (accumulate into a variable checked after the run); vary inputs; JIT can still specialize, so benchmark at the macro level for important claims
Python	Less optimization risk, but keep the call identical to real usage (no cached results across rounds)

• Results that are suspiciously fast (sub-nanosecond per op, identical for all sizes) mean the work was eliminated. Fix the guard before trusting anything.

4. Quiet machine and A/A calibration

• Linux benchmark host: performance CPU governor, turbo disabled, pinned cores (taskset -c 2), nothing else running. Python: python3 -m pyperf system tune (pyperf system reset afterwards).
• Laptop: AC power, close browsers and IDE indexers, let it cool between long runs. Expect several percent noise regardless.
• A/A run: benchmark the same build twice and compare. The spread you see is the smallest difference you can detect. If A/A shows “significant” differences, the setup is too noisy for the question; fix the environment or switch to instruction counts.

5–6. Compare

Use the harness’s comparator, or the bundled script for any tool that exports raw samples:

python3 scripts/compare_bench.py base.txt new.txt --threshold 5 --alpha 0.01
python3 scripts/compare_bench.py old.json new.json --format pytest     # pytest-benchmark --benchmark-json
python3 scripts/compare_bench.py old.txt new.txt --format go           # go test -bench output

It prints median, change, bootstrap 95% CI, Mann-Whitney p-value, and relative spread per benchmark, and exits 1 when a change is both significant and larger than the threshold in the bad direction (--higher-is-better for throughput metrics). It flags fewer than 10 samples and spread above half the threshold.

Decision rules:

• ~ in benchstat, or p >= alpha: no detectable difference. Say so; do not report the raw percentage as a win.
• Significant but below the threshold you care about: real but not worth complexity.
• Significant and above threshold: confirm with a second independent run or a profile before claiming it in a PR.

Tool recipes

Go

func BenchmarkParse(b *testing.B) {
    input := loadFixture(b, "testdata/orders.json")   // outside the timed loop
    for b.Loop() {
        if _, err := Parse(input); err != nil {
            b.Fatal(err)
        }
    }
}

go install golang.org/x/perf/cmd/benchstat@latest
git stash && go test -run '^$' -bench 'Parse' -benchmem -count 10 ./pkg/... > old.txt
git stash pop && go test -run '^$' -bench 'Parse' -benchmem -count 10 ./pkg/... > new.txt
benchstat old.txt new.txt

JMH

@State(Scope.Benchmark)
@BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS)
@Fork(3) @Warmup(iterations = 5, time = 1) @Measurement(iterations = 10, time = 1)
public class ParseBench {
  @Param({"100", "10000"}) int size;
  String payload;
  @Setup public void setup() { payload = Fixtures.orders(size); }
  @Benchmark public Order parse() { return Parser.parse(payload); }   // returned -> not eliminated
}

Run with java -jar target/benchmarks.jar ParseBench -rf json -rff results.json; add -prof gc for allocation rates. Never set @Fork(0) for real measurements.

pytest-benchmark and pyperf

def test_parse(benchmark, orders_payload):
    result = benchmark(parse, orders_payload)
    assert result.total > 0

pytest tests/bench --benchmark-only --benchmark-autosave           # saves .benchmarks/...
pytest tests/bench --benchmark-only --benchmark-compare --benchmark-compare-fail=median:5%
pytest tests/bench --benchmark-only --benchmark-json=new.json      # raw samples for compare_bench.py
python3 -m pyperf timeit -s 'from app import parse, P' 'parse(P)' -o new.json
python3 -m pyperf compare_to old.json new.json --table

Keep benchmarks out of the normal test run (--benchmark-skip in default addopts).

Rust (Criterion)

use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

fn bench_parse(c: &mut Criterion) {
    let input = std::fs::read_to_string("benches/orders.json").unwrap();
    c.bench_function("parse orders", |b| b.iter(|| parse(black_box(&input))));
}
criterion_group!(benches, bench_parse);
criterion_main!(benches);

Cargo.toml: [[bench]] name = "parse" harness = false. Compare branches with cargo bench -- --save-baseline main on main, then cargo bench -- --baseline main on the branch. divan is lighter (#[divan::bench(args = [...])], divan::main()); use it when compile time and simple syntax matter more than Criterion’s reports.

Vitest

Vitest 5 benchmarks live in *.bench.ts files and use the bench fixture:

import { test, expect } from "vitest";

test("parser comparison", async ({ bench }) => {
  const result = await bench.compare(
    bench("JSON.parse", () => JSON.parse(payload)),
    bench("customParse", () => customParse(payload)),
  );
  expect(result.get("customParse")).toBeFasterThan(result.get("JSON.parse"));
});

Run with vitest bench. On Vitest 4 or earlier, the API is import { bench, describe } from "vitest" with vitest bench; check vitest --version before writing either form. Standalone: tinybench new Bench({ time: 500 }), .add(name, fn), await bench.run(), console.table(bench.table()).

CI regression gates

Approach	Signal stability	Use when
Instruction counts (Cachegrind-based: Gungraun for Rust, CodSpeed’s simulation mode)	High; independent of runner noise	Gating PRs on shared CI runners
Wall clock, base and head in the same job, interleaved	Medium	Self-hosted or dedicated runners
Wall clock against stored history (benchmark-action/github-action-benchmark)	Low on shared runners	Trend dashboards; alert, do not block

Rules:

• Set the threshold from A/A runs on the same runner type: at least 2 to 3 times the observed spread. Wall-clock noise on shared hosted runners is often larger than the regressions you care about; measure yours rather than assuming.
• github-action-benchmark defaults alert-threshold to 200%; lower it only as far as your measured noise allows, and set fail-on-alert only for stable signals.
• Run benchmarks on PRs that touch hot paths (path filters) and nightly on main.
• A failing gate posts the comparison table and the rerun command, so a human can confirm before blocking a merge.

Gotchas

• Benchmarking the cache: the same input every iteration hits warm CPU caches and memoization; real calls do not. Rotate inputs from a pool.
• Setup inside the timed loop (JSON fixture load, regex compile, DB connect).
• Comparing a debug build to a release build, or JIT-cold to JIT-warm runs.
• GC pauses landing in some samples: report allocations (-benchmem, -prof gc) next to time; fewer allocations often matter more than the time delta.
• Micro win, macro loss: a faster function that allocates more can slow the service. Confirm important wins with a macro benchmark or a load test.
• Thermal throttling on laptops in long runs makes later samples slower; interleave A and B.
• Mean of 3 runs reported as “12% faster” with no spread.

Verification

1. Sanity: the benchmark result scales as expected with input size; a deliberately slowed version (add a sleep or extra loop) is detected as a regression.
2. A/A comparison shows no significant difference.
3. The A/B comparison meets the decision rules above, with sample counts stated.
4. For CI gates, a synthetic regression commit trips the gate and a no-op commit does not.

Output contract

## Benchmark: <question>
- Harness: <tool + version>; machine: <CPU, OS, governor/power>; build: <flags>
- Inputs: <sizes/fixtures>
- Samples: <n> per side; A/A spread: <x>%
| Benchmark | Base median | New median | Change | 95% CI | p | Verdict |
|---|---|---|---|---|---|---|
- Allocations: <B/op, allocs/op before -> after>
- Conclusion: <significant improvement of x% | no detectable change | regression>; confirmed by <profile/macro run>
- Reproduce: <exact commands>