Fine-Tuning skill for Claude Code

Fine-tuning is justified when a prompted (and, if needed, retrieval-augmented) baseline has been measured and plateaued, and a tuned model beats it on a held-out eval at an acceptable cost to train and serve. A fine-tune that was never compared with a good prompt on the same eval is an expensive guess.

Should you fine-tune?

Goal	Better first choice	Fine-tune when
Model lacks knowledge of your documents	RAG (rag-pipelines)	Almost never; fine-tuning teaches knowledge poorly and it goes stale
Consistent format, style, or tone	Prompt + examples, structured outputs	Prompt is long and costly per call, or format still drifts at scale
Narrow classification / extraction at high volume	Prompted small model	Need lower latency/cost than any prompted model gives at the target accuracy
Domain-specific behavior a strong model does with a long prompt	Prompt caching	You want a small self-hosted model to match it (distillation)
Teach a new skill the base model cannot do at all	Bigger model	Rarely works with small data; expect thousands of high-quality examples
Data cannot leave your infrastructure	Local prompted model (local-llms)	Prompted local model falls short of the bar

Hosted fine-tuning availability differs by provider and model and changes often; check the provider’s current documentation. This skill covers open-weight models, where you control the whole pipeline.

Workflow

- [ ] 1. Baseline: best prompt (+ RAG) on the eval set; record score and cost
- [ ] 2. Build and clean the dataset; split train/val/test
- [ ] 3. Pick base model and method (LoRA, QLoRA, full)
- [ ] 4. Train with conservative hyperparameters, watching val loss
- [ ] 5. Evaluate on the held-out test set vs. baseline and base model
- [ ] 6. Merge or serve adapters; re-run eval on the serving stack

2. Dataset

Format: chat-style JSONL, one conversation per line, matching how you will call the model.

{"messages": [{"role": "system", "content": "You classify support tickets."}, {"role": "user", "content": "My card was charged twice"}, {"role": "assistant", "content": "{\"category\": \"billing\", \"urgency\": \"high\"}"}]}

Item	Guidance
Size	200-1,000 excellent examples for style/format; 1K-10K for narrow tasks; quality beats quantity
Source	Real inputs; outputs written or reviewed by experts, or generated by a stronger model and filtered
Coverage	Match the production distribution, plus edge cases and “decline” examples
Cleaning	Deduplicate (exact and near-duplicate), remove PII, fix inconsistent labels, drop truncated outputs
Splits	Train/val/test 80/10/10 by source or customer, so near-duplicates do not leak into test
Consistency	Same system prompt and format as inference; label guidelines written down

Read 50 random examples before training. Every inconsistent label teaches the model to be inconsistent.

Preference data (for DPO): {"prompt": ..., "chosen": ..., "rejected": ...} with real, meaningful differences; collect from human comparisons or graded model samples.

3. Base model and method

• Start from the instruct/chat variant of a current open model family in the 1-8B range for narrow tasks; go larger only if the eval demands it.
• Check the license permits your use.

Method	Memory	Use when
LoRA (bf16 base)	Base weights + small adapters	Default on a GPU that fits the base model in bf16
QLoRA (4-bit base)	~1/4 of bf16 for the base	Limited GPU memory; small quality cost
Full fine-tune	Weights + gradients + optimizer state (~16 bytes/param with AdamW)	Large data, maximum quality, multi-GPU budget

4. Train (TRL + PEFT)

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = "your-org/base-instruct-model"
ds = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)          # QLoRA; omit for LoRA
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb,
                                             dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

peft_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                         target_modules="all-linear", task_type="CAUSAL_LM")

args = SFTConfig(
    output_dir="out/ticket-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,          # effective batch 16
    learning_rate=2e-4,                     # LoRA; use ~1e-5 to 2e-5 for full fine-tunes
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    gradient_checkpointing=True,
    eval_strategy="steps", eval_steps=50,
    save_strategy="steps", save_steps=50,
    load_best_model_at_end=True, metric_for_best_model="eval_loss",
    logging_steps=10,
    assistant_only_loss=True,               # train on assistant turns, not prompts
    report_to="none",
)
trainer = SFTTrainer(model=model, args=args, train_dataset=ds["train"],
                     eval_dataset=ds["validation"], processing_class=tokenizer,
                     peft_config=peft_config)
trainer.train()
trainer.save_model("out/ticket-lora/final")

TRL and transformers rename arguments between releases (torch_dtype became dtype, evaluation_strategy became eval_strategy, tokenizer became processing_class). Check the installed versions (pip show trl transformers peft) and their docs when an argument is rejected. Loss masking options also differ by version; confirm only assistant tokens contribute to loss.

Hyperparameter defaults and when to move them:

Knob	Default	Move when
LoRA rank r	16 (alpha = 2r)	8 for tiny datasets; 32-64 for harder tasks with more data
Learning rate	1e-4 to 2e-4 (LoRA)	Lower if val loss spikes or outputs degrade
Epochs	1-3	Stop at the val-loss minimum; more epochs on small data memorizes
Max sequence length	p99 of your examples	Truncation silently cuts assistant answers

For preference tuning after SFT, TRL’s DPOTrainer with beta around 0.1 and a lower learning rate (5e-7 to 5e-6 for full; ~5e-5 for LoRA) is the usual starting point.

5. Evaluate

• Compare three systems on the same held-out test set: base model + best prompt, the fine-tuned model, and the original strong prompted baseline (possibly a hosted model).
• Also run a small general-capability and safety check: fine-tuning can erase instruction following or refusals outside the task.
• Inspect outputs for memorized training strings (exact-match search against train set).
• Use llm-evals for graders and paired comparison.

6. Serve

• Merge LoRA into the base for simple serving (model.merge_and_unload() on a bf16 base, not a 4-bit one), or serve adapters dynamically (vLLM and others support multiple LoRA adapters on one base).
• Use the exact chat template from training at inference. Template mismatch is the most common cause of “worked in the notebook, garbage in production”.
• Re-run the eval through the actual serving stack, including any quantization (GGUF, AWQ) applied for deployment.

Gotchas

• Training loss going down proves nothing; val loss rising while train loss falls means overfitting. Keep the best checkpoint by val metric.
• Training on prompt tokens (no loss masking) wastes capacity and can make the model echo inputs.
• Missing EOS token at the end of assistant turns causes the tuned model to ramble past the answer.
• Pad token set to EOS with right-padding during training but left-padding needed for batched generation; configure both explicitly.
• Contaminated test sets (near-duplicates of training examples) inflate scores. Split by source and deduplicate across splits.
• Fine-tuning on outputs from a hosted model may be restricted by that provider’s terms; check before distilling.

Output contract

**Decision:** fine-tune because <baseline score/cost plateau evidence>
**Data:** <n> train / <n> val / <n> test, sources, cleaning steps, label guide link
**Model/method:** <base model, license>, <LoRA r/alpha | QLoRA | full>, <lr, epochs, seq len>
**Results (test):** baseline prompt <a>, base+prompt <b>, fine-tuned <c>; general check <pass/fail>
**Serving:** <merged | adapter>, <quantization>, eval on serving stack <score>