LLM Evals skill for Claude Code
Know whether the change helped, with error bars. Claude builds evals that can actually decide things. Cases come from real traffic and past failures, each criterion gets the cheapest reliable grader, and any LLM judge is checked against human labels before its scores count. Version comparisons are paired on the same cases with a McNemar test and bootstrap interval, and a bundled script fails CI when must-pass cases regress.
- name
llm-evals - size173 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Builds evaluation suites for LLM features - sourcing test cases from real traffic, choosing code, model, or human graders, writing LLM-as-judge rubrics that agree with humans, running paired comparisons between prompt or model versions, and gating regressions in CI. Use when someone asks whether a prompt or model change helped, before switching models, when building an LLM judge, when outputs regress silently, or when an AI feature needs a quality bar.Try asking
- Did switching our summarizer to the smaller model make it worse? Prove it either way
- Build an LLM judge for faithfulness and check that it agrees with our reviewers
- Set up an eval that blocks prompt changes in CI if quality regresses
Use it when
- You need to know whether a prompt or model change helped or hurt
- You are about to switch models or reasoning effort levels
- You are building or trusting an LLM-as-judge grader
- An AI feature needs a quality bar enforced before release
Skip it when
- You are measuring a retriever's recall rather than answers — start with rag-pipelines
- You need dashboards of live traffic quality and cost — use llm-observability
How to check the result
The judge shows high agreement with human labels on both passes and failures, and scripts/compare_runs.py reports the paired difference with a confidence interval on the held-out split.
The common mistake it prevents
Trusting a 1-to-10 judge score that was never compared with human labels, so the eval rewards longer answers rather than better ones.
--- name: llm-evals description: Builds evaluation suites for LLM features - sourcing test cases from real traffic, choosing code, model, or human graders, writing… license: MIT ---
An eval is trustworthy when its cases come from real usage, its graders have been checked against human judgment, comparisons between versions are paired on the same cases with a noise estimate, and a regression fails CI. A single aggregate score from an unvalidated judge on synthetic prompts measures the judge, not the product.
Ground rules
- Look at the data before building graders. Read 50+ real outputs and write down the failure modes you actually see; those become criteria.
- Prefer the cheapest grader that is reliable: code > model > human.
- One criterion per grader call. “Rate overall quality 1-10” is unusable.
- Compare versions on the same cases (paired), and report the uncertainty.
- Never tune a prompt on the test split. Keep a held-out set you do not read while iterating.
- Every eval run spends money; estimate cost (cases x calls x tokens) before running large sweeps and tell the user.
Workflow
- [ ] 1. Define what the feature must do and must never do
- [ ] 2. Collect and label cases (dev split + held-out test split)
- [ ] 3. Pick a grader per criterion
- [ ] 4. Validate model graders against human labels
- [ ] 5. Build the runner (deterministic, cached, logs everything)
- [ ] 6. Compare versions with paired statistics
- [ ] 7. Gate in CI and grow the set from production failures1. Criteria
Write criteria as observable, binary checks where possible:
- “Response cites at least one provided source” (not “is well grounded”).
- “Does not recommend a specific medication dose” (safety, must-never).
- “Extracted
invoice_totalequals the labeled value within 0.01.”
Split into must-pass (any failure blocks release) and quality (tracked as a rate).
2. Cases
| Source | Use for | Caveat |
|---|---|---|
| Production logs (sampled, PII-scrubbed) | Realistic distribution | Needs labeling; sample failures and thumbs-down heavily |
| Support tickets / bug reports | Known failures | Becomes the regression set |
| Hand-written edge cases | Boundaries, adversarial inputs, empty/huge inputs | Over-represents what you already imagined |
| Model-generated variations | Coverage of phrasings and languages | Review by hand; filter duplicates and nonsense |
Sizes: 50-100 cases detects large effects (10+ points); 300-500 detects ~5-point
changes. Split 70/30 dev/test, stratified by category. Store as JSONL with an id,
input, expected (or labels), category, and source.
3. Pick graders
| Criterion type | Grader | Example |
|---|---|---|
| Exact facts, classification labels, JSON fields | Code: equality, regex, schema validation | pred["label"] == gold["label"] |
| Numeric extraction | Code with tolerance | abs(pred - gold) <= 0.01 |
| Runs / compiles / tests pass | Code: execute in a sandbox | unit tests on generated code |
| Faithfulness, tone, instruction following, helpfulness | Model judge with rubric | binary pass/fail with reason |
| Brand, legal, subtle domain correctness | Human review | weekly sample of 50 |
LLM-as-judge rubric
JUDGE_SYSTEM = """You grade one criterion for an AI assistant's answer.
Criterion: Every factual claim in the answer is supported by the provided documents.
Pass: all claims are supported, or the answer correctly says the documents lack the answer.
Fail: any claim is absent from or contradicted by the documents.
Think through each claim in <analysis> tags, then output <verdict>PASS</verdict> or <verdict>FAIL</verdict>."""
def judge(documents, question, answer):
resp = client.messages.create(
model="claude-opus-5-5", # judge at least as strong as the model under test
max_tokens=2000,
system=JUDGE_SYSTEM,
messages=[{"role": "user", "content":
f"<documents>{documents}</documents>\n<question>{question}</question>\n<answer>{answer}</answer>"}],
)
text = "".join(b.text for b in resp.content if b.type == "text")
m = re.search(r"<verdict>(PASS|FAIL)</verdict>", text)
return {"pass": m and m.group(1) == "PASS", "raw": text, "parsed": bool(m)}Judge rules:
- Binary verdicts with a definition of each side beat 1-10 scales. If you need gradations, use 3-4 labeled levels with an anchor example each.
- Reasoning before the verdict improves agreement; parse only the verdict tag. Count unparseable outputs as errors, not passes.
- For pairwise comparisons (A vs B), run both orders and only count consistent wins; judges favor the first or longer response.
- Do not let the judge see which version produced an answer.
- A model judging its own outputs tends to be lenient; use a different or stronger model where possible, and always validate.
4. Validate the judge
Label 50-100 cases by hand (two humans if possible). Compute agreement between judge and humans: report accuracy plus true-positive rate and true-negative rate separately (a judge that passes everything has high accuracy on a 90%-pass set). Target >= 85-90% agreement and that the judge catches most human-labeled failures. Iterate on the rubric, not the threshold, until it does. Re-validate when you change the judge model or rubric.
5. Runner
- Pin everything: model ID, prompt version, parameters, grader version, dataset hash. Write them into the results file header.
- Cache model outputs by (input hash, config hash) so re-grading does not re-generate.
- Run with bounded concurrency and the SDK’s retries; record errors as a separate status, never as fail or pass.
- Sampling is nondeterministic, and several current Claude models reject non-default
temperature, sotemperature=0is not an option. For noisy criteria, run each case 3 times and grade the pass rate per case. - Output one JSONL line per case:
{"id", "pass", "score", "category", "output", "grader_raw"}. - Use the provider’s batch API for large offline runs (about half price on Anthropic).
6. Compare versions
python3 scripts/compare_runs.py results_v1.jsonl results_v2.jsonl --max-drop 0.02The script pairs cases by id, prints pass rates per version and per category, lists
regressions (pass -> fail) and fixes (fail -> pass), computes an exact McNemar p-value and a
bootstrap 95% CI for the difference, and exits 1 if v2 is worse than v1 by more than
--max-drop or significantly worse at p < 0.05.
Decision rules:
- CI excludes zero in the good direction and no must-pass regressions: ship.
- CI includes zero: no evidence of change; prefer the cheaper or simpler version.
- Any must-pass case flips to fail: block, regardless of the aggregate.
- Read every regression case before deciding; aggregate gains can hide a new failure mode.
7. CI and growth
- Run a fast smoke subset (20-50 must-pass cases) on every prompt/model change; run the full set nightly or before release.
- Every production incident or thumbs-down that reveals a new failure becomes a case.
- Rebalance when the set drifts from real traffic (track category mix).
Gotchas
- Synthetic-only eval sets are too easy; scores look great and production disagrees.
- Overfitting: after 20 iterations on the same 50 cases, the dev score is meaningless. Report the held-out score as the headline.
- Judge drift: upgrading the judge model changes scores with no product change. Pin it.
- Length bias: judges reward longer answers. Add “length is not a quality signal” and check with a length-controlled comparison.
- Averaging across categories hides regressions in small but important ones; always show the per-category table.
- Leaking expected answers into the prompt under test (shared templates, few-shot examples drawn from the test set) inflates scores.
Output contract
**Eval:** <name> — <n> cases (<dev>/<test>), sources <logs/tickets/hand-written>
**Graders:** <criterion -> code|model|human>; judge agreement with humans <x>% (TPR <a>, TNR <b>)
**Comparison:** v1 <p1>% -> v2 <p2>% (diff <d>, 95% CI [<lo>, <hi>], McNemar p=<p>)
**Regressions:** <count> — <ids and one-line cause each>
**Decision:** ship | hold | revert — <reason>
**Cost of run:** <$ or tokens>Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/compare_runs.py
#!/usr/bin/env python3
"""Paired comparison of two LLM eval runs.
Usage:
python3 compare_runs.py baseline.jsonl candidate.jsonl [--max-drop 0.02] [--alpha 0.05]
[--bootstrap 2000] [--seed 0] [--must-pass-key must_pass]
Each line: {"id": "case-1", "pass": true, "category": "billing", "must_pass": false}
Lines with "pass": null (or a "status": "error") are reported as errors and excluded from rates.
Exits 1 when the candidate is worse than baseline by more than --max-drop, when the
drop is significant (McNemar exact p < alpha), or when any must-pass case regresses.
Exits 2 on input errors.
"""
import argparse
import json
import random
import sys
from collections import defaultdict
from math import comb
def load(path):
rows = {}
with open(path, encoding="utf-8") as fh:
for n, line in enumerate(fh, 1):
line = line.strip()
if not line:
continue
try:
r = json.loads(line)
except json.JSONDecodeError as exc:
print(f"error: {path}:{n}: {exc}", file=sys.stderr)
sys.exit(2)
if "id" not in r:
print(f"error: {path}:{n}: missing id", file=sys.stderr)
sys.exit(2)
rows[str(r["id"])] = r
return rows
def outcome(r):
if r.get("status") == "error" or r.get("pass") is None:
return None
return bool(r["pass"])
def mcnemar_exact(b, c):
"""Two-sided exact binomial test on discordant pairs."""
n = b + c
if n == 0:
return 1.0
k = min(b, c)
tail = sum(comb(n, i) for i in range(0, k + 1)) / 2 ** n
return min(1.0, 2 * tail)
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("baseline")
ap.add_argument("candidate")
ap.add_argument("--max-drop", type=float, default=0.02)
ap.add_argument("--alpha", type=float, default=0.05)
ap.add_argument("--bootstrap", type=int, default=2000)
ap.add_argument("--seed", type=int, default=0)
ap.add_argument("--must-pass-key", default="must_pass")
args = ap.parse_args()
a, b = load(args.baseline), load(args.candidate)
shared = sorted(set(a) & set(b))
only_a, only_b = set(a) - set(b), set(b) - set(a)
if only_a or only_b:
print(f"warning: {len(only_a)} ids only in baseline, {len(only_b)} only in candidate (ignored)")
pairs, errors = [], []
for cid in shared:
oa, ob = outcome(a[cid]), outcome(b[cid])
if oa is None or ob is None:
errors.append(cid)
continue
pairs.append((cid, oa, ob))
if not pairs:
print("error: no comparable cases", file=sys.stderr)
sys.exit(2)
n = len(pairs)
pa = sum(x[1] for x in pairs) / n
pb = sum(x[2] for x in pairs) / n
regress = [cid for cid, oa, ob in pairs if oa and not ob]
fixed = [cid for cid, oa, ob in pairs if ob and not oa]
p = mcnemar_exact(len(regress), len(fixed))
rng = random.Random(args.seed)
diffs = []
for _ in range(args.bootstrap):
sample = [pairs[rng.randrange(n)] for _ in range(n)]
diffs.append(sum(s[2] for s in sample) / n - sum(s[1] for s in sample) / n)
diffs.sort()
lo = diffs[int(0.025 * len(diffs))]
hi = diffs[int(0.975 * len(diffs)) - 1]
print(f"cases compared: {n} (errors excluded: {len(errors)})")
print(f"baseline pass rate: {pa:.3f}")
print(f"candidate pass rate: {pb:.3f}")
print(f"difference: {pb - pa:+.3f} 95% bootstrap CI [{lo:+.3f}, {hi:+.3f}] McNemar p={p:.4f}")
print(f"regressions (pass->fail): {len(regress)} fixes (fail->pass): {len(fixed)}")
cats = defaultdict(lambda: [0, 0, 0])
for cid, oa, ob in pairs:
c = str(b[cid].get("category", a[cid].get("category", "uncategorized")))
cats[c][0] += 1
cats[c][1] += oa
cats[c][2] += ob
print("\nper category: n baseline candidate diff")
for c, (cn, sa, sb) in sorted(cats.items()):
print(f" {c:<22} {cn:>4} {sa / cn:.3f} {sb / cn:.3f} {(sb - sa) / cn:+.3f}")
if regress:
print("\nregressed ids:", ", ".join(regress[:30]) + (" ..." if len(regress) > 30 else ""))
must = [cid for cid in regress if b[cid].get(args.must_pass_key) or a[cid].get(args.must_pass_key)]
failed = False
if must:
print(f"\nFAIL: {len(must)} must-pass case(s) regressed: {', '.join(must[:10])}")
failed = True
if pa - pb > args.max_drop:
print(f"FAIL: pass rate dropped {pa - pb:.3f} > max-drop {args.max_drop}")
failed = True
if pb < pa and p < args.alpha:
print(f"FAIL: significant regression (p={p:.4f} < {args.alpha})")
failed = True
if failed:
sys.exit(1)
verdict = "improved" if lo > 0 else "no detectable change"
print(f"\nPASS: {verdict}")
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.