Flaky Test Triage skill for Claude Code
Find flakes from CI data and fix the cause. Claude treats flakiness as a measurable defect. It aggregates JUnit reports from many CI runs to find tests that flip, ranks them by pipeline cost, quarantines the worst into a non-blocking job with an owner, reproduces them under repeat, shuffle and CPU stress, and only calls a fix done when the failure rate drops to zero over an equal number of runs.
- name
flaky-test-triage - size149 lines · ≈1,750 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Detects, measures, quarantines, and fixes flaky tests in CI using data instead of reruns. Covers mining JUnit reports across runs for pass/fail flips, reproducing with repeat and shuffle flags in Playwright, Jest, Vitest, pytest, and Go, classifying root causes (async waits, shared state, ordering, time, network, resources), a quarantine policy with owners, and proving a fix by failure rate. Use when CI is red intermittently, when retries hide failures, when a test passes locally but fails in CI, or when setting up a flaky-test process for a team.Try asking
- CI fails randomly about once a day. Find which tests are flaky from these reports
- This Playwright test passes locally but fails in GitHub Actions sometimes
- Set up a quarantine process for flaky tests on our team
Use it when
- CI goes red intermittently without code changes
- Retries are hiding failures and slowing pipelines
- A test passes locally but fails in CI
- Designing a team policy for quarantining flaky tests
Skip it when
- The failure is deterministic and reproducible every time: use systematic-debugging
- The flakiness is only in screenshot diffs: use visual-regression
How to check the result
The reproduction loop that showed failures now passes with zero failures over at least as many runs, and the quarantine job stays green for the agreed window before release.
The common mistake it prevents
Adding a sleep or another retry so the test stops failing, which hides the race while it keeps costing pipeline time and sometimes masks a real product bug.
A flaky test is a test whose result changes without a code change. Done well: flakes are found from data, ranked by cost, quarantined with an owner within a day, fixed at the root cause, and released from quarantine only after a measured run of passes. Blanket retries are a painkiller, not a cure.
Ground rules
- Never “fix” a flake by adding a sleep, a longer timeout, or another retry without knowing the mechanism.
- A fix is proven only by failure rate: the same number of repeated runs that showed failures must now show zero.
- Quarantine is not deletion. Quarantined tests keep running in a non-blocking job so the failure rate stays visible.
- Sometimes the flake is a real product bug (race in production code). Check before blaming the test.
Workflow
- [ ] 1. Collect evidence: which tests flip, how often, where
- [ ] 2. Rank by cost (failure rate x pipeline runs x time lost)
- [ ] 3. Quarantine top offenders with owner and issue
- [ ] 4. Reproduce locally or in CI with repeat/shuffle/parallel stress
- [ ] 5. Classify the mechanism and fix the root cause
- [ ] 6. Prove the fix by failure rate; release from quarantine
- [ ] 7. Prevent: lint rules, isolation defaults, retry reporting1. Collect evidence
Download JUnit XML reports from the last N CI runs (artifacts) and aggregate them:
python3 scripts/junit_flakes.py ci-reports/ # directory of XML files from many runs
python3 scripts/junit_flakes.py run-*/junit.xml --min-runs 5 --json > flakes.jsonThe script lists tests that both passed and failed across runs (or were retried within a run), with failure rate, retry counts, duration spread, and the most common failure message; it separates consistently failing tests, which are real failures. It exits 1 if any flaky test is found.
Other data sources: test reporters’ built-in flaky markers (Playwright reports tests that passed on retry as “flaky”), CI analytics (GitHub Actions re-run counts, Buildkite or CircleCI test insights), and “re-run failed jobs” frequency on main.
2. Rank
Cost ≈ failure rate x runs per day x (pipeline duration + human time per re-run). A 2% flake in a test that runs 300 times a day blocks six pipelines daily. Fix the top three first; long tails can wait.
3. Quarantine
| Framework | Mechanism |
|---|---|
| Playwright | Tag { tag: '@quarantine' }; main job --grep-invert @quarantine, separate non-blocking job --grep @quarantine |
| Jest / Vitest | Move to a quarantine project or use a naming convention excluded from the blocking run; run it in a non-blocking job |
| pytest | @pytest.mark.quarantine (registered marker); blocking job -m "not quarantine" |
| Go | Build tag or t.Skip guarded by an env var, run unskipped in a nightly job |
| JUnit 5 | @Tag("quarantine") and exclude it in the blocking build |
Every quarantine entry gets: an issue link, an owner, a date, and the measured failure rate. Set a time limit (e.g., two weeks) after which it escalates.
4. Reproduce
Loop the single test first, then add stress:
npx playwright test e2e/cart.spec.ts --repeat-each=50 --workers=4 --retries=0
npx vitest run src/cart.test.ts --sequence.shuffle --retry=0 # loop in shell for counts
npx jest src/cart.test.ts --runInBand # vs. parallel to detect shared state
pytest tests/test_cart.py --count=50 -x # needs pytest-repeat
pytest -p no:randomly tests/ # compare with pytest-randomly on
go test ./cart -run TestCheckout -count=200 -raceShell loop for any runner:
fails=0; for i in $(seq 1 50); do npm test -- cart >/dev/null 2>&1 || fails=$((fails+1)); done; echo "$fails/50 failed"If it never fails locally, reproduce CI conditions: same container image, CPU limits
(docker run --cpus=2), same parallelism, same test ordering or shard, same time zone
(TZ=UTC), cold caches. Capture artifacts on failure (traces, logs, DB dumps).
5. Classify and fix
| Experiment result | Mechanism | Real fix |
|---|---|---|
| Fails under CPU load or parallelism | Missing await, race, timing assumption | Await the operation; wait on an observable condition; web-first assertions |
| Fails only after a specific other test | Shared state leak (globals, DB rows, mocks, env) | Reset in afterEach; per-test transaction or unique data; restore mocks |
| Fails only with shuffle | Order dependence | Make each test create its own preconditions |
| Fails around midnight/month-end/DST or in other TZ | Real clock | Inject or freeze time |
| Fails with different seed | Unseeded randomness | Seed and log the seed; constrain generated data |
| Fails with parallel workers only | Shared port, file, DB schema, account | Port 0, temp dirs per test, schema/account per worker |
| Fails when network is slow | Real external calls | Stub at HTTP boundary |
| Timeouts on CI only | Resource starvation, slow cold start | Profile; reduce work; raise timeout only with evidence |
| Product race found | Real concurrency bug | Fix production code; keep the test as the regression guard |
E2E-specific: replace waitForTimeout with expect(locator).toBeVisible() or
page.waitForResponse; avoid asserting on transient states (spinners) that may finish
before the check; make test data unique per test (e2e-${testInfo.testId}).
6. Prove and release
- Re-run the reproduction loop with at least as many iterations as the one that failed (for a 2% flake, 200+ runs gives reasonable confidence).
- Release from quarantine after the fix merges and the non-blocking job shows zero failures for an agreed window (e.g., 50 consecutive CI runs).
7. Prevent
- Default to
retries: 0locally; in CI allow at most 1–2 retries and report every pass-on-retry as flaky (fail the build on new flakes if the tooling supports it). - Lint: forbid
waitForTimeout,sleepin tests, focused tests (forbidOnly), and unawaited promises (@typescript-eslint/no-floating-promises). - Isolation defaults:
restoreMocks: true, per-test DB transactions, random test order enabled in CI.
Gotchas
- Retrying at the job level hides flakes from test-level reporting entirely.
- A flake that disappears when you add logging is timing-sensitive; use low-overhead tracing instead of breakpoints.
- Infrastructure flakes (runner OOM, registry timeouts) look like test flakes. Check whether failures cluster by runner or step, not by test.
- Fixing the symptom in the test while the product has a real race ships the race.
Output
**Flake:** CartPage › applies coupon (e2e/cart.spec.ts) — 7/300 CI runs failed (2.3%)
**Mechanism:** assertion ran before POST /api/coupon resolved; button re-rendered and the
locator matched a detached element.
**Fix:** await the response and assert on the updated total with a web-first assertion.
**Proof:** `--repeat-each=300 --workers=6` → 0 failures (was 6/300).
**Quarantine:** released after 50 green non-blocking runs. Issue #1234 closed.Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/junit_flakes.py
#!/usr/bin/env python3
"""Find flaky tests across many JUnit XML reports.
Usage:
python3 scripts/junit_flakes.py <report-or-dir> [<report-or-dir> ...] [--min-runs 3] [--json]
Point it at JUnit XML files from several CI runs of the same commit range (or a
directory tree of them). A test is reported as flaky when it both passed and failed
across the runs, or when a report records an in-run retry (Surefire-style
<flakyFailure>/<flakyError>/<rerunFailure>/<rerunError> elements).
Exit codes: 0 = no flaky tests found, 1 = flaky tests found, 2 = no parsable reports.
"""
import argparse
import json
import os
import sys
import xml.etree.ElementTree as ET
from collections import Counter, defaultdict
RETRY_TAGS = {"flakyFailure", "flakyError", "rerunFailure", "rerunError"}
def iter_files(paths):
for p in paths:
if os.path.isdir(p):
for root, _, files in os.walk(p):
for f in files:
if f.endswith(".xml"):
yield os.path.join(root, f)
elif os.path.isfile(p):
yield p
def first_line(text):
text = (text or "").strip()
return text.splitlines()[0][:160] if text else ""
def parse(path, stats):
try:
tree = ET.parse(path)
except ET.ParseError as exc:
print(f"warn: skipping {path}: {exc}", file=sys.stderr)
return False
for case in tree.getroot().iter("testcase"):
tid = f"{case.get('classname') or case.get('file') or '?'}::{case.get('name')}"
s = stats[tid]
tags = {child.tag for child in case}
if "skipped" in tags:
s["skipped"] += 1
continue
failed = [c for c in case if c.tag in ("failure", "error")]
if failed:
s["failed"] += 1
msg = first_line(failed[0].get("message") or failed[0].text)
if msg:
s["messages"][msg] += 1
else:
s["passed"] += 1
retries = [c for c in case if c.tag in RETRY_TAGS]
if retries:
s["retried"] += len(retries)
msg = first_line(retries[0].get("message") or retries[0].text)
if msg:
s["messages"][msg] += 1
try:
s["time"].append(float(case.get("time") or 0))
except ValueError:
pass
return True
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("paths", nargs="+")
ap.add_argument("--min-runs", type=int, default=2, help="ignore tests seen in fewer runs (default 2)")
ap.add_argument("--json", action="store_true", help="print machine-readable output")
args = ap.parse_args()
stats = defaultdict(lambda: {"passed": 0, "failed": 0, "skipped": 0, "retried": 0,
"messages": Counter(), "time": []})
parsed = sum(parse(f, stats) for f in iter_files(args.paths))
if not parsed:
print("error: no parsable JUnit XML reports found", file=sys.stderr)
sys.exit(2)
flaky, broken = [], []
for tid, s in stats.items():
runs = s["passed"] + s["failed"]
if runs < args.min_runs and not s["retried"]:
continue
if (s["passed"] and s["failed"]) or s["retried"]:
rate = s["failed"] / runs if runs else 0.0
flaky.append((tid, s, runs, rate))
elif s["failed"] and not s["passed"]:
broken.append((tid, s, runs))
flaky.sort(key=lambda x: (-x[3], -x[1]["retried"], x[0]))
if args.json:
out = {
"reports": parsed,
"flaky": [{"test": t, "runs": r, "failed": s["failed"], "retried": s["retried"],
"failure_rate": round(rate, 4),
"top_message": (s["messages"].most_common(1) or [("", 0)])[0][0]}
for t, s, r, rate in flaky],
"consistently_failing": [t for t, _, _ in broken],
}
print(json.dumps(out, indent=2))
else:
print(f"Parsed {parsed} report(s), {len(stats)} distinct test(s).\n")
if flaky:
print(f"FLAKY ({len(flaky)}): failure rate across runs, in-run retries, top message")
for tid, s, runs, rate in flaky:
msg = (s["messages"].most_common(1) or [("", 0)])[0][0]
times = s["time"]
spread = f"{min(times):.2f}-{max(times):.2f}s" if times else "-"
print(f" {rate:6.1%} {s['failed']}/{runs} runs retries={s['retried']} time={spread} {tid}")
if msg:
print(f" {msg}")
else:
print("No flaky tests detected.")
if broken:
print(f"\nCONSISTENTLY FAILING ({len(broken)}) - real failures, not flakes:")
for tid, s, runs in broken:
print(f" {s['failed']}/{runs} runs {tid}")
sys.exit(1 if flaky else 0)
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.