Model Serving skill for Claude Code

A serving deployment is ready when a load test at expected peak traffic, with realistic prompt and output lengths, meets the latency SLO (time to first token and inter-token latency at p95) with headroom, the cost per million tokens is known, and autoscaling has been exercised. Picking a model and running vllm serve is the first 5%.

Ground rules

• Size from numbers: weights + KV cache + overhead must fit with margin. Guessing GPU counts leads to OOM at peak or idle money.
• Measure with your traffic shape. Throughput for 200-token prompts says nothing about 8k-token RAG prompts.
• Pin versions: engine, CUDA image, model revision (commit hash), tokenizer, chat template.
• Quantize only after measuring quality on your eval; a quantization that breaks tool calling or JSON output is not a saving.
• Put an authenticated gateway in front. Engine API keys are a minimum, not access control.

Choose the engine

Situation	Engine
General production GPU serving, broadest model support	vLLM
Heavy prefix reuse (agents, multi-turn, shared system prompts), structured generation at scale	SGLang (benchmark against vLLM on your workload)
CPU, Apple Silicon, small GPUs, single-tenant edge boxes	llama.cpp llama-server with GGUF
Existing Hugging Face TGI deployment	Plan a migration: TGI upstream is in maintenance mode and recommends vLLM or SGLang
Kubernetes platform with many models, routing, scale-to-zero	An orchestration layer (KServe, llm-d, NVIDIA Dynamo, or similar) on top of vLLM/SGLang

Single-developer or offline local use belongs in local-llms; this skill is about shared, multi-user endpoints.

Workflow

- [ ] 1. Define the SLO and traffic model
- [ ] 2. Size memory and pick GPUs + parallelism
- [ ] 3. Choose precision / quantization and verify quality
- [ ] 4. Configure the engine
- [ ] 5. Load test; tune batching knobs
- [ ] 6. Deploy behind a gateway with metrics and health checks
- [ ] 7. Autoscale on queue depth; test scale-up time
- [ ] 8. Record cost per million tokens and the runbook

1. SLO and traffic model

Write down: peak requests per second, prompt length distribution (p50/p95), output length distribution, concurrency, streaming or not, and the SLO, for example “TTFT p95 under 800 ms, inter-token latency p95 under 50 ms, error rate under 0.1%”. Get lengths from real logs if they exist.

2. Memory sizing

weights_bytes  = params x bytes_per_param        (bf16=2, fp8=1, int4 ~0.5-0.6)
kv_per_token   = 2 x layers x kv_heads x head_dim x kv_bytes
kv_needed      = kv_per_token x (concurrent_seqs x avg_total_tokens)
gpu_budget     = gpu_mem x gpu_memory_utilization - weights - activations/overhead (~1-3 GB)

Worked example, Llama-3.1-8B-class (32 layers, 8 KV heads, head dim 128, bf16 KV): 2 x 32 x 8 x 128 x 2 = 128 KiB per token. 64 concurrent sequences at 6k tokens each need ~48 GiB of KV cache on top of ~16 GB of weights: one 80 GB GPU fits; a 24 GB GPU does not. A 70B-class model (80 layers, 8 KV heads) is ~320 KiB per token in bf16.

Weights do not fit one GPU	Use tensor parallelism across GPUs in one node (2, 4, 8)
Fits one GPU, need more throughput	Data parallelism: more replicas (or --data-parallel-size)
Does not fit one node	Pipeline parallelism across nodes; only when unavoidable
KV cache is the limit	FP8 KV cache, shorter max context, fewer max sequences, or more GPUs

Read the model’s config.json (num_hidden_layers, num_key_value_heads, head_dim or hidden_size / num_attention_heads) rather than trusting blog numbers.

3. Precision and quantization

Option	When	Watch for
BF16	Quality baseline; memory is available	Most memory
FP8 weights (and FP8 KV cache)	Hopper/Blackwell-class GPUs; usually the best default to try	Check accuracy on your eval, especially long-context
AWQ / GPTQ int4	Memory-bound, older GPUs, larger model on fewer GPUs	Quality drops on reasoning, code, and tool calls; use published checkpoints with eval results
GGUF Q4_K_M / Q5_K_M / Q8_0	llama.cpp deployments	Q4 is noticeably worse on small models

Run your eval set (see llm-evals) on each candidate; keep the cheapest that holds quality, including structured-output validity and tool-call accuracy.

4. Configure the engine

vLLM (OpenAI-compatible server):

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --revision <commit-sha> \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --kv-cache-dtype fp8 \
  --served-model-name llama-70b \
  --api-key "$VLLM_API_KEY" \
  --port 8000

Knob	Effect	Default stance
--max-model-len	Caps context; directly bounds KV per sequence	Set to what traffic needs, not the model maximum
--max-num-seqs	Max sequences batched at once	Raise for throughput; lower if TTFT/ITL suffer
--max-num-batched-tokens	Tokens per scheduler step (prefill chunk size)	Larger favors throughput, smaller favors ITL
--gpu-memory-utilization	Fraction of GPU memory vLLM claims	0.85-0.92; leave room if other processes share the GPU
Prefix caching	Reuses KV for shared prompt prefixes	On (default in current vLLM); keep system prompts byte-stable
--enable-auto-tool-choice + --tool-call-parser	Tool calling via the OpenAI API	Required for tool use; parser must match the model family

SGLang: python -m sglang.launch_server --model-path <model> --tp 4 --port 30000. llama.cpp: llama-server -m model.gguf -c 32768 -np 4 -ngl 99 --port 8080; the context is split across parallel slots, so -c 32768 -np 4 gives each slot 8k tokens.

Flags change between releases; check vllm serve --help for the installed version.

5. Load test

vllm bench serve --backend openai-chat --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions --model llama-70b \
  --dataset-name random --random-input-len 2000 --random-output-len 300 \
  --num-prompts 1000 --request-rate 8

Or replay real prompts with any load tool that streams and records TTFT (see load-testing). Sweep request rate until the SLO breaks; the knee is your per-replica capacity. Record: output tokens/s, requests/s, TTFT p50/p95, ITL p95, and KV cache usage at the knee. Plan peak traffic at ~70% of knee capacity.

6. Deploy

• Health: liveness on the process, readiness only after the model is loaded and a warm-up request succeeded (loading 70B weights takes minutes).
• Gateway in front: auth, per-tenant rate limits and token quotas, request size limits, routing by model name, and logging of usage per tenant.
• Clients use the OpenAI-compatible API (/v1/chat/completions), so swapping engines or falling back to a hosted API is a base-URL change.
• Scrape /metrics with Prometheus. Key vLLM metrics: vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:e2e_request_latency_seconds, and vllm:prefix_cache_hits / vllm:prefix_cache_queries for hit rate. Metric names have changed across versions; confirm against your engine’s /metrics output.
• Alerts: waiting requests above N for 2+ minutes, TTFT p95 over SLO, KV usage pinned near 1.0 with preemptions, 5xx rate, GPU XID errors.

7. Autoscaling

• Scale on queue depth (num_requests_waiting) or KV cache usage, not CPU. GPU utilization is a poor signal; it is high even at comfortable load.
• On Kubernetes, KEDA with a Prometheus trigger or a custom-metrics HPA works; set a stabilization window so replicas do not flap.
• Scale-up time = node provisioning + image pull + weight download + load + warm-up. Measure it. Cut it with pre-pulled images, weights cached on local NVMe or a shared volume, and a minimum replica count that covers baseline traffic.
• Scale-to-zero only for internal or batch endpoints where a multi-minute cold start is acceptable.

8. Cost

cost per 1M output tokens = (GPU $/hour x GPUs) / (output tokens/s at target load x 3600) x 1e6. Compare with hosted API prices for the same quality; self-hosting wins at steady high utilization or under data-residency constraints, and often loses at spiky low volume.

Gotchas

• Chat template mismatches silently degrade quality; serve with the model’s own template and test a few prompts against reference outputs.
• Setting --max-model-len to the model’s 128k maximum shrinks the number of concurrent sequences and can make startup fail when the KV cache cannot hold one full sequence.
• Long prompts block short ones during prefill; chunked prefill and token budgets help, separate pools for long-context traffic help more.
• Mixed GPU types in one tensor-parallel group run at the speed of the slowest.
• A shared GPU (another process grabbing memory) causes OOM at startup; dedicate GPUs.
• Model licenses differ (Llama community license, Gemma terms, Apache-2.0, research-only); confirm commercial use before deploying.

Output contract

**Model:** <name@revision>, precision <bf16/fp8/awq>, eval delta vs bf16 <x>
**Engine:** <vLLM/SGLang/llama.cpp> <version>, TP <n>, max_model_len <n>, max_num_seqs <n>
**Hardware:** <GPU x count> per replica, replicas min <n> / max <n>
**Load test:** knee at <req/s>, TTFT p95 <ms>, ITL p95 <ms>, <tok/s> output
**Autoscaling:** metric <name> target <value>, measured scale-up <minutes>
**Cost:** <$ per 1M output tokens> at <utilization>%