Skill Description Tuning skill for Claude Code
Make skills fire on the right prompts, and only those. Treats a skill description as a routing classifier and tests it like one. Claude builds a labeled set of realistic prompts and near-miss negatives, runs each in a fresh headless session to see whether the skill actually fires, diagnoses the miss pattern, and rewrites one variable at a time until recall is high, false positives are zero, and sibling skills still win their own requests.
- name
skill-description-tuning - size175 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Diagnoses Agent Skills that never trigger, trigger too often, or lose to a sibling skill, then rewrites the description against a labeled set of should-trigger and should-not-trigger prompts run in fresh headless sessions. Use when a skill does not load when expected, loads on unrelated requests, the wrong skill fires, or before publishing a skill whose description has never been tested.Try asking
- My deploy skill never loads unless I type /deploy — figure out why
- The testing skill keeps firing on every code question, tighten it
- Write trigger tests for this skill's description before I publish it
Use it when
- A skill works when invoked by name but not when the task is described
- A skill loads on requests it has nothing to do with
- Two skills compete and the wrong one wins
- You want evidence a description works before sharing the skill
Skip it when
- You are creating a skill from nothing — start with skill-authoring
- The skill loads correctly but its instructions are weak — edit the body instead
How to check the result
Rerun the full labeled prompt set in fresh sessions: positives load the skill, near-miss negatives do not, and held-out prompts behave the same way.
The common mistake it prevents
Testing in the same session where the skill was written, so its body is already in context and every prompt appears to trigger it.
--- name: skill-description-tuning description: Diagnoses Agent Skills that never trigger, trigger too often, or lose to a sibling skill, then rewrites the description against a labeled… license: MIT ---
A description is tuned when a labeled prompt set passes in fresh sessions: at least 9 of 10 should-trigger prompts load the skill, zero near-miss prompts do, and no sibling skill regressed. Judging by “it worked when I tried it once” is not tuning.
How routing works (what you are optimizing)
- Before any skill loads, Claude sees only each skill’s
nameanddescriptionin a listing. The body, references, and scripts are invisible at decision time. - Claude picks a skill when the request plausibly matches the description’s stated triggers. Keywords that appear only in the body do nothing for routing.
- Manual
/skill-nameinvocation bypasses the description entirely. A skill that works when typed but not when described is a description problem. - The total skill listing has a size budget. With many installed skills, long
descriptions cost everyone; check
/contextto see the listing’s weight.
Workflow
- [ ] 1. Confirm the skill registers at all
- [ ] 2. Build a labeled prompt set (positives + near-miss negatives)
- [ ] 3. Run a baseline and record the confusion matrix
- [ ] 4. Diagnose the failure pattern
- [ ] 5. Rewrite the description (one hypothesis per revision)
- [ ] 6. Re-run; check siblings; stop at the pass bar or after 3 rounds1. Confirm registration first
Run /skills. If the skill is missing, no description change will help:
| Symptom | Cause | Fix |
|---|---|---|
| Not listed | YAML error (often an unquoted : inside the description) | Quote the description or remove the colon |
| Not listed | name differs from folder, or uppercase/underscores in name | Rename to match the folder, lowercase-hyphen |
| Not listed | New top-level skills dir created mid-session | /reload-skills or restart |
| Listed, never auto-loads | disable-model-invocation: true | Remove it if model invocation is wanted |
| Listed, loads only in some dirs | paths restricts it | Widen or remove paths |
| A different skill with the same name loads | Precedence: enterprise > personal > project | Rename one of them |
2. Build the prompt set
Write it to a file so it can be rerun. 8-12 positives, 6-10 negatives:
[
{"prompt": "the orders query takes 4 seconds, can you speed it up", "expect": true},
{"prompt": "EXPLAIN shows a seq scan on events, what index do I need", "expect": true},
{"prompt": "review the indexes in migrations/0042_add_orders.sql", "expect": true},
{"prompt": "which of our postgres indexes are unused", "expect": true},
{"prompt": "design the schema for a new invoices table", "expect": false},
{"prompt": "the API is slow because it runs a query per row", "expect": false},
{"prompt": "set up a redis cache in front of the product endpoint", "expect": false}
]Positives must vary along these axes, or the test only proves the obvious case:
- Explicit (“use the indexing skill”) vs. implicit (describes the problem only).
- Casual, terse, and detailed phrasing; typos and lowercase.
- Pasted artifacts: an error string, a file path, a config snippet.
- The task embedded mid-request (“after the migration, also check the indexes”).
Negatives must be near misses: same domain words, different job (usually a sibling skill’s territory). “Write a poem” as a negative proves nothing.
Never put the skill’s name in more than one positive. Users rarely type it.
3. Run the baseline
Automated (requires the claude CLI on PATH and the skill installed where the run’s
working directory will see it):
python3 ${CLAUDE_SKILL_DIR}/scripts/trigger_eval.py cases.json --skill postgres-indexing --runs 3The script runs each prompt in a fresh claude -p session with stream-json output,
detects whether the target skill (or a different one) was invoked in the first turns,
prints a per-prompt table plus recall, false-positive count, and sibling hits, and exits
non-zero on any mismatch. Edit, Write, and Bash are disallowed during the run so test
prompts cannot change files; run it from a scratch checkout anyway. Each prompt costs a
short session, so keep sets small and use --runs 3 only on the cases that flip.
Manual fallback: open a new session per prompt (/clear is not enough if the skill was
already loaded earlier in the same session), send the prompt, and watch for the skill
being invoked before any other work.
4. Diagnose
| Pattern | Likely cause | Fix |
|---|---|---|
| Low recall, implicit prompts miss | Description names the solution (“applies indexing methodology”), not the problem users state | Add symptom phrases: “slow query”, “seq scan”, “takes N seconds” |
| Low recall, artifact prompts miss | No file types, tool names, or error strings | Name them: EXPLAIN, .sql migrations, pg_stat_user_indexes |
| Loads after Claude already started | Trigger phrasing is weak or buried at the end | Lead with the task verb; add “Use before writing X” when timing matters |
| False positives on generic requests | Generic nouns (“code”, “files”, “performance”) carry the description | Replace with domain-specific nouns; drop “any”/“all” |
| A sibling wins | Overlapping keywords, no boundary | Add “Not for <their job>; use <sibling>.” to both descriptions |
| Flips run to run | Description is borderline for that phrasing | Strengthen one concrete trigger; do not add five vague ones |
5. Rewrite
Formula:
<What it does: verb + concrete nouns + tools/file types>. <Scope, output, or method>.
Use when <user phrases>, <symptoms/errors>, <file types or situations>. [Not for <X>; use <sibling>.]Before (never triggers on “this query is slow”):
Provides database optimization expertise and best practices for improving performance.After:
Designs and audits PostgreSQL indexes from EXPLAIN plans: B-tree, partial, covering,
GIN, and expression indexes, and finds unused ones safely. Use when a query is slow,
EXPLAIN shows a Seq Scan on a large table, a migration adds indexes, or the user asks
which indexes to add or drop. Not for N+1 query patterns in application code.Rules:
- Change one thing per revision (add symptoms, or add a boundary, not both), so the rerun tells you what worked.
- Harvest vocabulary from real prompts in the user’s history, issue titles, and error messages, not from a thesaurus.
- Stay within 1024 characters; 300-600 is usually enough. Every added word taxes the shared listing budget.
- Keep it plain text, third person, no markdown or XML.
- Claude Code also reads a
when_to_usefield. It is useful for local skills but is rejected by claude.ai uploads, so portable skills keep triggers indescription.
6. Re-run and stop
- Rerun the full set, not only the failures. Fixes for recall often create false positives.
- If a boundary sentence was added, rerun the sibling skill’s set too.
- Stop when the pass bar is met. After three rounds without convergence, the skill’s scope is probably wrong: split it, or merge it with the sibling it keeps colliding with.
- Hold out 2-3 prompts you never tune against and check them last to catch overfitting.
Gotchas
- Testing inside the session where you wrote the skill: the body is already in context, so everything looks like it triggers.
- Tuning against prompts that all contain the skill’s own keywords. Real users describe symptoms.
- Treating a single run as truth. Routing is probabilistic; borderline cases need 3 runs.
- Fixing over-triggering by making the description vague. Narrow it with specific nouns and a “Not for” clause instead.
- Editing the body to fix routing. Only the description participates in routing.
Output
**Skill:** postgres-indexing
**Prompt set:** 10 positive, 8 negative (2 held out)
| Round | Recall | False positives | Sibling hits | Change |
|---|---|---|---|---|
| 0 | 5/10 | 1/8 | n-plus-one-queries x1 | baseline |
| 1 | 8/10 | 1/8 | 0 | added symptom phrases + EXPLAIN/seq scan |
| 2 | 10/10 | 0/8 | 0 | added "Not for N+1 queries" boundary |
**Held-out:** 2/2 correct
**Final description:** (paste)Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/trigger_eval.py
#!/usr/bin/env python3
"""Measure whether a skill triggers on a labeled prompt set, using headless Claude Code.
Usage:
python3 trigger_eval.py cases.json --skill NAME [--runs N] [--max-turns N]
[--timeout SECONDS] [--model MODEL] [--allow-side-effects]
cases.json is a list of {"prompt": str, "expect": bool}.
Each prompt runs in a fresh `claude -p` session with `--output-format stream-json`.
A case counts as triggered when the stream contains a Skill tool call naming the target
skill, or a Read of .../<skill>/SKILL.md. Other skills invoked are reported as sibling
hits. With --runs N a case is "triggered" when the majority of runs triggered.
By default Edit, Write, NotebookEdit and Bash are disallowed so test prompts cannot
change files. Exit code 0 when every case matches its expectation, 1 otherwise.
Standard library only.
"""
from __future__ import annotations
import argparse
import json
import shutil
import subprocess
import sys
def scan(obj, skill: str, hits: set[str]) -> bool:
"""Walk a stream-json event looking for tool_use blocks."""
found = False
if isinstance(obj, dict):
if obj.get("type") == "tool_use":
name = obj.get("name", "")
inp = obj.get("input", {}) or {}
blob = json.dumps(inp)
if name == "Skill":
invoked = str(inp.get("skill") or inp.get("command") or inp.get("name") or blob)
invoked = invoked.split(":")[-1].strip("/ ")
if invoked == skill or f'"{skill}"' in blob:
found = True
else:
hits.add(invoked)
elif name == "Read" and f"/{skill}/SKILL.md" in blob:
found = True
for v in obj.values():
found = scan(v, skill, hits) or found
elif isinstance(obj, list):
for v in obj:
found = scan(v, skill, hits) or found
return found
def run_once(prompt: str, args) -> tuple[bool, set[str], str]:
cmd = ["claude", "-p", prompt, "--output-format", "stream-json", "--verbose",
"--max-turns", str(args.max_turns)]
if args.model:
cmd += ["--model", args.model]
if not args.allow_side_effects:
cmd += ["--disallowedTools", "Edit", "Write", "NotebookEdit", "Bash"]
try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=args.timeout)
except subprocess.TimeoutExpired:
return False, set(), "timeout"
hits: set[str] = set()
triggered = False
for line in proc.stdout.splitlines():
line = line.strip()
if not line.startswith("{"):
continue
try:
event = json.loads(line)
except json.JSONDecodeError:
continue
triggered = scan(event, args.skill, hits) or triggered
note = "" if proc.stdout.strip() else (proc.stderr.strip().splitlines() or ["no output"])[-1][:80]
return triggered, hits, note
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("cases")
ap.add_argument("--skill", required=True)
ap.add_argument("--runs", type=int, default=1)
ap.add_argument("--max-turns", type=int, default=2)
ap.add_argument("--timeout", type=int, default=180)
ap.add_argument("--model")
ap.add_argument("--allow-side-effects", action="store_true")
args = ap.parse_args()
if not shutil.which("claude"):
print("ERROR: `claude` CLI not found on PATH")
return 2
with open(args.cases, encoding="utf-8") as f:
cases = json.load(f)
if not isinstance(cases, list) or not all("prompt" in c and "expect" in c for c in cases):
print("ERROR: cases file must be a list of {\"prompt\": ..., \"expect\": true|false}")
return 2
tp = fn = fp = tn = 0
sibling_total: dict[str, int] = {}
print(f"{'result':8} {'expect':6} {'rate':5} prompt")
for case in cases:
prompt, expect = case["prompt"], bool(case["expect"])
triggered_runs = 0
hits_all: set[str] = set()
notes = []
for _ in range(args.runs):
trig, hits, note = run_once(prompt, args)
triggered_runs += trig
hits_all |= hits
if note:
notes.append(note)
triggered = triggered_runs * 2 > args.runs
ok = triggered == expect
if expect and triggered: tp += 1
elif expect: fn += 1
elif triggered: fp += 1
else: tn += 1
for h in hits_all:
sibling_total[h] = sibling_total.get(h, 0) + 1
extra = f" [other: {', '.join(sorted(hits_all))}]" if hits_all else ""
extra += f" [{notes[0]}]" if notes else ""
label = "ok" if ok else "MISMATCH"
print(f"{label:8} {str(expect):6} {triggered_runs}/{args.runs:<3} {prompt[:70]}{extra}")
pos, neg = tp + fn, fp + tn
print()
print(f"Recall: {tp}/{pos}" + (f" ({tp / pos:.0%})" if pos else ""))
print(f"False positives: {fp}/{neg}")
if sibling_total:
print("Sibling hits: " + ", ".join(f"{k} x{v}" for k, v in sorted(sibling_total.items())))
passed = fn == 0 and fp == 0
print(f"Result: {'PASS' if passed else 'FAIL'}")
return 0 if passed else 1
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.