Hugging Face Transformers skill for Claude Code

Hub-based code is right when the model revision is pinned, weights load in the intended dtype on the intended device, the tokenizer and chat template match the model, batched inference uses correct padding and attention masks, and outputs are checked against a known-good sample. Most “the model is bad” reports are template, padding, or dtype bugs.

Ground rules

• Pin model revisions (revision="<commit sha>") in anything that ships. Repos change under the same name.
• Load safetensors weights. Avoid pickle .bin files from untrusted repos.
• Never set trust_remote_code=True without reading the repo’s Python files; it executes them.
• Use the model’s own tokenizer and chat template. Do not hand-format chat prompts.
• Check library versions (pip show transformers datasets accelerate) before copying examples; argument names change between releases.

Workflow

- [ ] 1. Pick the model: task, license, size, revision
- [ ] 2. Download and cache deliberately (HF_HOME, token, offline mode)
- [ ] 3. Load with explicit dtype and device placement
- [ ] 4. Tokenize / template correctly; verify on one example
- [ ] 5. Batch inference or fine-tune
- [ ] 6. Validate outputs against a reference sample

1-2. Pick and download

pip install -U transformers accelerate datasets huggingface_hub safetensors
hf auth login                                   # or export HF_TOKEN=... (needed for gated models)
hf download <org>/<model> --revision <sha> --include "*.safetensors" "*.json" "tokenizer*" \
  --local-dir ./models/<model>
export HF_HOME=/data/hf                         # cache location; default ~/.cache/huggingface
export HF_HUB_OFFLINE=1                         # in production: never hit the network at load time

Older docs use huggingface-cli; the current CLI is hf. Check the model card for license, intended use, context length, and the recommended generation settings.

3. Load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "<org>/<model>"
tok = AutoTokenizer.from_pretrained(model_id, revision=REV)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=REV,
    dtype=torch.bfloat16,       # older releases: torch_dtype=
    device_map="auto",          # needs accelerate; spreads across GPUs, spills to CPU
)

Situation	Loading choice
GPU with enough memory	dtype=torch.bfloat16, device_map="auto"
Not enough GPU memory	4-bit or 8-bit via BitsAndBytesConfig, or a pre-quantized (AWQ/GPTQ) checkpoint
CPU only, small model	float32 (bf16 on CPU is slow on many chips)
Apple Silicon	device_map="mps" or .to("mps"); or use MLX instead (see local-llms)
Production serving, many users	Do not serve with generate() loops; use vLLM or SGLang

Check placement: print(model.hf_device_map) and model.dtype. Layers on cpu or disk explain slow generation.

4. Tokenization and chat templates

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "Explain KV caching in two sentences."},
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

• add_generation_prompt=True appends the assistant header; without it the model may continue the user turn.
• Decode only the new tokens (slice off the prompt length).
• Print the templated string once (tokenize=False) to confirm special tokens and roles.
• Some models do not support a system role; the template raises or silently drops it. Merge it into the first user message if so.

5a. Batched generation

tok.padding_side = "left"                       # decoder-only models must pad on the left
if tok.pad_token is None:
    tok.pad_token = tok.eos_token
prompts = [tok.apply_chat_template([{"role": "user", "content": q}], add_generation_prompt=True,
                                   tokenize=False) for q in questions]
batch = tok(prompts, return_tensors="pt", padding=True, add_special_tokens=False).to(model.device)
with torch.inference_mode():
    out = model.generate(**batch, max_new_tokens=128, pad_token_id=tok.pad_token_id)
texts = tok.batch_decode(out[:, batch["input_ids"].shape[1]:], skip_special_tokens=True)

• Right padding in decoder-only generation produces garbage after the prompt.
• Pass the attention mask (**batch does); missing masks cause warnings and wrong output.
• add_special_tokens=False when the template already inserted BOS, or you get two.
• Sort inputs by length and batch similar lengths to cut padding waste.

Pipelines (quick tasks)

from transformers import pipeline

clf = pipeline("text-classification", model="<org>/<classifier>", revision=REV, device_map="auto")
clf(["great product", "arrived broken"], batch_size=32, truncation=True)

Good for classification, NER, summarization, ASR, and quick prototypes. For custom control (logits, batching strategy, streaming), use the model and tokenizer directly.

Embeddings

from sentence_transformers import SentenceTransformer

emb_model = SentenceTransformer("<org>/<embedding-model>", revision=REV)
vecs = emb_model.encode(texts, batch_size=64, normalize_embeddings=True, show_progress_bar=True)

Use the model card’s query/document prefixes if it defines them (many retrieval models expect "query: " / "passage: " style prefixes). Store vectors per vector-search.

5b. Datasets

from datasets import load_dataset

ds = load_dataset("csv", data_files={"train": "train.csv", "test": "test.csv"})
# or load_dataset("<org>/<dataset>", revision=...), streaming=True for huge corpora

def tokenize(batch):
    return tok(batch["text"], truncation=True, max_length=512)

ds = ds.map(tokenize, batched=True, num_proc=8, remove_columns=["text"])
split = ds["train"].train_test_split(test_size=0.1, seed=42)

• batched=True is many times faster than per-row map.
• map results are cached on disk keyed by function hash; changing a closure variable without changing code can reuse a stale cache. Pass load_from_cache_file=False when in doubt.
• Do not pad in map; pad per batch with a data collator.

5c. Fine-tune a classifier with Trainer

from transformers import (AutoModelForSequenceClassification, DataCollatorWithPadding,
                          Trainer, TrainingArguments)
import numpy as np

labels = ["billing", "bug", "other"]
model = AutoModelForSequenceClassification.from_pretrained(
    base_id, num_labels=len(labels),
    id2label=dict(enumerate(labels)), label2id={l: i for i, l in enumerate(labels)})

def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=-1)
    return {"accuracy": (preds == p.label_ids).mean()}

args = TrainingArguments(output_dir="out/clf", learning_rate=2e-5, num_train_epochs=3,
                         per_device_train_batch_size=32, eval_strategy="epoch",
                         save_strategy="epoch", load_best_model_at_end=True,
                         metric_for_best_model="accuracy", bf16=True, report_to="none")
trainer = Trainer(model=model, args=args, train_dataset=split["train"], eval_dataset=split["test"],
                  processing_class=tok, data_collator=DataCollatorWithPadding(tok),
                  compute_metrics=compute_metrics)
trainer.train()

For LLM fine-tuning with LoRA, use fine-tuning. For custom loops, pytorch-training.

Publish

model.push_to_hub("<org>/<name>", private=True)
tok.push_to_hub("<org>/<name>", private=True)

Write a model card: base model, data, intended use, eval results, limitations, license.

6. Validate

• Run 5-10 inputs whose correct outputs you know (from the model card, a reference implementation, or the hosted inference widget) and compare.
• For batched code, confirm batch outputs equal one-at-a-time outputs for the same inputs (greedy decoding).
• Log transformers.__version__, model revision, dtype, and device map with results.

Gotchas

• Recent major releases are PyTorch-first (TensorFlow and Flax support was removed in v5); argument renames include torch_dtype -> dtype, evaluation_strategy -> eval_strategy, tokenizer= -> processing_class= in Trainer.
• fp16 on models trained in bf16 can overflow to NaN/inf outputs; prefer bf16 where the hardware supports it.
• max_length in generate counts prompt plus output; use max_new_tokens.
• Default generation settings come from the repo’s generation_config.json (sampling, temperature); set do_sample explicitly for reproducible tests.
• Truncating from the right on long inputs cuts the question at the end; for QA-style inputs truncate the context, not the query.
• Gated models fail with 401/403 until you accept the license on the Hub and use a token.

Output contract

**Model:** <org/model>@<revision>, license <x>, dtype <x>, device map <summary>
**Pipeline:** <load -> template -> batch size -> generation settings>
**Validation:** <n> reference inputs matched; batch == single: <yes/no>
**Versions:** transformers <v>, torch <v>, datasets <v>