Synthetic Data skill for Claude Code

Synthetic data earns its place when a model trained or evaluated with it does better on real held-out data than without it. Volume, fluency, and an LLM judge’s approval do not prove that. The usual failure is a large, fluent, repetitive dataset that teaches the generator’s habits instead of the task.

Ground rules

• Always keep a real held-out set as the final judge. Never evaluate on synthetic data generated by the same pipeline you are tuning.
• Diversity is designed, not requested. “Make them diverse” yields near-duplicates; explicit attribute grids and real seeds yield variety.
• Every row carries provenance: synthetic: true, generator model, prompt version, seed attributes, and timestamp.
• Filter hard. Expect to discard 20-60% of raw generations.
• Check provider terms before using model outputs to train other models, and never seed generation with raw personal data you are not allowed to process.

Decide whether synthetic is the right tool

Goal	Synthetic helps?	Better alternative if not
Bootstrap an eval set before launch	Yes, as a starting point; replace with real traffic ASAP	Log real queries from a beta
Cover rare classes or edge cases	Yes, targeted generation	Active sampling from production logs
Fine-tune a small model to copy a large one (distillation)	Yes, with real inputs and model-generated outputs	-
Replace real training data entirely	Rarely; quality ceiling is the generator	Label real data (see llm-classification)
Test fixtures and demo data	Yes; deterministic factories may be simpler	test-data-factories
Privacy-safe copy of a production database	Only with formal privacy methods; LLM paraphrase is not anonymization	Masking, differential privacy tooling

Workflow

- [ ] 1. Define the target distribution and the real held-out set
- [ ] 2. Build the diversity plan (attribute grid + real seeds)
- [ ] 3. Generate with structured output and provenance
- [ ] 4. Validate (schema, rules, execution)
- [ ] 5. Score quality (LLM judge calibrated against humans)
- [ ] 6. Deduplicate and check contamination
- [ ] 7. Compare distributions with real data
- [ ] 8. Prove usefulness on real held-out data

1. Target distribution

Write a short spec: task, input types, length range, languages, label set and target proportions, difficulty mix, and what “correct” means. Collect at least 50-200 real examples; they become seeds, calibration data for the judge, and the held-out set (split them; never seed from the held-out part).

2. Diversity plan

Build an attribute grid and sample combinations rather than asking for variety:

import itertools, random
GRID = {
    "persona": ["new user", "power user", "admin", "non-native speaker", "frustrated customer"],
    "topic": ["billing", "login", "data export", "integrations", "performance"],
    "difficulty": ["simple", "multi-part", "ambiguous", "contains a false premise"],
    "format": ["one line", "long email", "bullet list", "pasted error log"],
}
combos = list(itertools.product(*GRID.values()))
random.Random(7).shuffle(combos)
plan = [dict(zip(GRID, c)) for c in combos[:2000]]

Additional techniques:

• Seeded generation: give 3-5 real examples per call, rotated, and ask for new items in the same distribution but different content.
• Two-stage: first generate scenarios (short descriptions), dedupe them, then expand each into a full item. Deduping at the scenario level is cheaper and more effective.
• Hard negatives: for classifiers and retrieval, generate near-miss items that should get a different label or should not match.
• Long-tail weighting: oversample rare attribute values on purpose, then reweight to the target proportions when assembling the final set.

3. Generate

import anthropic, json
from pydantic import BaseModel

client = anthropic.Anthropic()

class Item(BaseModel):
    user_message: str
    expected_label: str
    rationale: str

def generate(attrs: dict, seeds: list[str]) -> Item:
    resp = client.messages.parse(
        model="claude-sonnet-5-5",
        max_tokens=1500,
        system=GENERATOR_SPEC,                       # task spec + label definitions, cached
        messages=[{"role": "user", "content":
            "Real examples for style and distribution only, do not copy them:\n"
            + "\n---\n".join(seeds)
            + f"\n\nWrite one new item with these attributes: {json.dumps(attrs)}"}],
        output_format=Item,
    )
    return resp.parsed_output

• Generate one item per call (or a small list) with explicit attributes; large single calls drift toward repetition after the first dozen items.
• For thousands of items, submit through a batch API (see llm-batch-jobs), keyed by the plan index so every row maps back to its attributes.
• When generating labeled data, have the generator target a label, then verify the label independently in step 5. Generator-assigned labels are wrong more often than expected.

4. Validate

Deterministic checks first, they are free:

• Schema validity, length bounds, language detection, banned content.
• Label in the allowed set; required fields present.
• For code: it parses, compiles, and its tests pass in a sandbox.
• For SQL or API calls: executes against a fixture database or mock.
• For math or extraction answers: recompute or cross-check against the source.

5. Score quality

Use an LLM judge with a rubric (realism, correctness of the label or answer, adherence to attributes). Calibrate it first: have a human score 50-100 items and confirm the judge agrees (see llm-evals). Use a different model or at least a different prompt from the generator to reduce self-preference. Drop items below the threshold and read a random sample of what passes.

6. Deduplicate and check contamination

python3 scripts/dedupe.py synthetic.jsonl --field user_message \
  --threshold 0.8 --against eval.jsonl --against-field question --out clean.jsonl

The script removes exact duplicates (after normalization) and near-duplicates (word shingle Jaccard via MinHash, default 3-word shingles, threshold 0.8), and flags rows whose 8-gram overlap with any item in --against exceeds --contam-threshold (default 0.5). It writes kept rows to --out and prints counts, the duplicate rate, and example clusters. A duplicate rate above ~15% means the diversity plan is too narrow.

Contamination matters in two directions: synthetic training data must not contain eval items, and an eval set generated by a model may already be memorized by that model family; prefer human-written or production-sourced eval items for final decisions.

7. Distribution comparison

Compare synthetic vs real on simple statistics before training anything:

Statistic	Red flag
Length distribution (median, p90)	Synthetic much more uniform or longer than real
Label proportions	Off from the target spec
Vocabulary / type-token ratio	Synthetic noticeably lower (repetitive)
Top n-grams	Generator tics (“I hope this helps”, “Certainly”, same opening phrase)
Embedding clusters	Synthetic occupies a few tight clusters; real data spreads wider

8. Prove usefulness

• Training data: train with real only, synthetic only, and real + synthetic; evaluate all on the real held-out set. Keep synthetic only if real + synthetic beats real only. Typical useful mixes put synthetic at 20-50% of the data; more often hurts.
• Eval data: check that model rankings on the synthetic eval match rankings on a small real eval. If they disagree, the synthetic eval is measuring something else.

Gotchas

• Generators default to polite, well-formed text. Real users write typos, fragments, mixed languages, and pasted logs; ask for these explicitly via attributes.
• Training repeatedly on a model’s own outputs narrows the distribution over generations. Keep real data in every training mix.
• Paraphrasing real records containing PII does not anonymize them; names, rare facts, and combinations remain identifying.
• Judges prefer outputs from their own model family. Use a different judge or human spot checks.
• Reusing the same few seed examples in every call copies their quirks into the whole dataset. Rotate seeds.

Output contract

**Purpose:** <eval / training / edge cases / fixtures>
**Real data:** <n> seeds, <n> held out
**Plan:** grid <attributes>, <n> combos, generator <model>, prompt v<n>
**Funnel:** generated <n> -> valid <n> -> judged <n> -> deduped <n> (dup rate <x>%, contaminated <y>)
**Distribution check:** <notes>
**Usefulness:** real-only <metric> vs real+synthetic <metric> on real held-out