Local LLMs skill for Claude Code
Size it, quantize it, benchmark it, then serve. Claude sizes memory before anything downloads, using a bundled estimator that reads the model's config to compute weights plus KV cache for your context and concurrency. It then picks a runtime and quantization, sets context length and GPU offload explicitly, benchmarks prefill and decode speed, checks quality on your own tasks and keeps the endpoint on localhost or behind auth.
- name
local-llms - size209 lines · ≈2,000 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Runs open-weight LLMs locally and on self-hosted servers with Ollama, llama.cpp, vLLM, and MLX - sizing memory for weights and KV cache, choosing GGUF quantization levels, setting context length, enabling GPU offload, exposing OpenAI-compatible endpoints, and benchmarking speed and quality. Use when someone wants to run a model offline or on-prem, picks hardware for local inference, gets out-of-memory errors or slow tokens per second, or needs a local drop-in endpoint for an app.Try asking
- What is the biggest model I can run on a 24 GB GPU with 32K context?
- Ollama is painfully slow on my machine. Figure out why
- Set up a local OpenAI-compatible endpoint so our app can run offline
Use it when
- You need to run a model offline, on-prem, or without sending data out
- You are choosing hardware or a quantization level for local inference
- Local inference is slow, runs out of memory, or ignores long prompts
- An app needs a local drop-in endpoint for development or privacy
Skip it when
- You are loading models in Python code for inference or training — use huggingface-transformers
- You want to adapt a model's weights to your data — use fine-tuning
How to check the result
The model loads with every layer on GPU, the observed memory is close to the estimate, decode speed meets the target, and quality on your own tasks holds against a reference.
The common mistake it prevents
Leaving the default context length in place, so long prompts are silently truncated and the model appears to ignore the system prompt.
--- name: local-llms description: Runs open-weight LLMs locally and on self-hosted servers with Ollama, llama.cpp, vLLM, and MLX - sizing memory for weights and KV cache,… license: MIT ---
A local deployment is right when the model fits entirely in fast memory with the needed context, the measured tokens per second meets the use case, quality was checked on your own tasks at the chosen quantization, and the endpoint is not exposed without authentication. “It runs” with half the layers on CPU at 3 tokens per second is not done.
Ground rules
- Size memory before downloading: weights + KV cache for the target context + overhead.
- Prefer a larger model at Q4 over a smaller model at Q8 for the same memory, but verify on your tasks.
- Set context length explicitly. Defaults are often small and prompts are silently truncated.
- Keep the server on localhost unless it sits behind a reverse proxy with auth and TLS.
- Benchmark on your own prompts. Leaderboard scores do not survive quantization equally.
Workflow
- [ ] 1. Define the job: model quality needed, context length, users/concurrency
- [ ] 2. Estimate memory and pick model size + quantization
- [ ] 3. Pick the runtime
- [ ] 4. Run with explicit context, offload, and KV settings
- [ ] 5. Benchmark speed and quality
- [ ] 6. Expose an OpenAI-compatible endpoint safely2. Size memory
python3 scripts/vram_estimate.py --params 8 --quant Q4_K_M --ctx 16384 --config config.json --vram 12The script reads layer and KV-head counts from the model’s Hugging Face config.json (or
--layers --kv-heads --head-dim), prints weights, KV cache, and total, and exits 1 if it
will not fit in --vram with 10% headroom.
Rules of thumb:
- Weights ~ params x bits / 8. 8B at Q4_K_M ~ 4.9 GB; 70B at Q4_K_M ~ 42 GB.
- KV cache grows linearly with context and concurrent sequences; at 32K+ context it can rival the weights.
- Apple Silicon: unified memory, but macOS reserves part of it for the system by default (roughly a quarter to a third); plan on ~65-75% of RAM for the model.
- Decode speed upper bound ~ memory bandwidth / model bytes. 400 GB/s and a 5 GB model: at most ~80 tokens/s. Prompt processing (prefill) is compute-bound and benefits from GPU.
Quantization choice (GGUF)
| Quant | Size vs F16 | Use when |
|---|---|---|
| Q8_0 | ~53% | Memory is plentiful; near-lossless reference |
| Q6_K / Q5_K_M | ~40% / ~36% | Quality-sensitive (code, math) and it fits |
| Q4_K_M | ~30% | Default balance for most models |
| IQ4_XS / Q3_K_M | ~25-27% | Squeezing a bigger model in; check quality |
| Q2_K / IQ2 | < 22% | Last resort; small models degrade badly |
Mixture-of-experts models need memory for all experts even though only a few run per token; they are fast for their size but not small.
3. Pick the runtime
| Situation | Runtime |
|---|---|
| Developer laptop, quick setup, model management | Ollama |
| Maximum control, CPU+GPU split, any GGUF, embedded | llama.cpp (llama-server) |
| Apple Silicon, best native speed, Python workflows | MLX (mlx-lm) |
| Multi-user GPU server, high throughput, batching | vLLM (or SGLang) with safetensors / AWQ / FP8 weights |
| GUI for non-developers | LM Studio (also serves an OpenAI-compatible API) |
4. Run
Ollama:
ollama pull llama3.1:8b
ollama run llama3.1:8b "Summarize: ..."
ollama ps # shows loaded models and the CPU/GPU split
ollama show llama3.1:8b # context length, parameters, templateServer settings (environment variables for ollama serve):
export OLLAMA_CONTEXT_LENGTH=16384 # default context for all models
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0 # halves KV memory; needs flash attention
export OLLAMA_NUM_PARALLEL=2 # concurrent requests per model (multiplies KV memory)
export OLLAMA_KEEP_ALIVE=30m # keep the model loaded between requestsPer-model defaults via a Modelfile:
FROM llama3.1:8b
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
SYSTEM You are a concise assistant for internal docs.ollama create docs-assistant -f Modelfilellama.cpp:
brew install llama.cpp # or build: cmake -B build -DGGML_CUDA=ON && cmake --build build -j
llama-server -hf <user>/<repo>-GGUF:Q4_K_M \
-c 16384 -ngl 99 --jinja \
--cache-type-k q8_0 --cache-type-v q8_0 -fa on \
--host 127.0.0.1 --port 8080-ngl 99offloads all layers to GPU; lower it only when the model does not fit (each layer left on CPU costs speed).--jinjauses the model’s embedded chat template, needed for correct tool calling.- Flash-attention flag syntax changed across builds (
-favs.-fa on); checkllama-server --helpfor your version. A quantized V cache requires flash attention. -np Nserves N parallel slots; the context budget is shared or per-slot depending on version, so confirm the per-slot context in the startup log.
MLX (Apple Silicon):
pip install mlx-lm
mlx_lm.generate --model mlx-community/<model>-4bit --prompt "Hello" --max-tokens 200
mlx_lm.server --model mlx-community/<model>-4bit --port 8080vLLM (Linux, NVIDIA/AMD GPU server):
vllm serve <org>/<model> --max-model-len 16384 --gpu-memory-utilization 0.90 \
--api-key "$LOCAL_LLM_KEY" --port 80005. Benchmark
Speed:
llama-bench -m model.gguf -p 512 -n 128 # prefill and decode tokens/s
ollama run llama3.1:8b --verbose "..." # prints eval rate after the responseTargets: interactive chat >= 15-20 tokens/s decode; coding assistants want 30+. Time to first token for a long prompt is dominated by prefill; measure it with your real prompt length.
Quality: run 30-50 of your own tasks through the local model and through a reference
(a hosted model or a higher-precision quant), and grade with llm-evals. For comparing
quants of the same model, llama-perplexity on representative text gives a fast signal.
6. Expose an endpoint
All runtimes above speak the OpenAI Chat Completions format, so apps can point any OpenAI-compatible client at them:
| Runtime | Base URL |
|---|---|
| Ollama | http://localhost:11434/v1 (native API at /api/chat) |
| llama-server | http://localhost:8080/v1 |
| vLLM | http://localhost:8000/v1 |
Keep model choice behind your app’s client module so you can swap between a local model and a hosted API (for example the Anthropic SDK for Claude) by configuration.
Security:
- Ollama and llama-server have no authentication by default. Never bind
0.0.0.0on an untrusted network; put nginx or Caddy with auth and TLS in front. - Download weights from the original publisher or well-known quantizers; check licenses.
- Hugging Face models that need
trust_remote_code=Trueexecute arbitrary Python; read the code first.
Gotchas
- Silent context truncation: the prompt exceeds the configured context and the runtime drops the beginning. Symptoms: model ignores the system prompt or early documents.
- Partial offload cliff: speed drops several-fold once layers spill to CPU. Check
ollama psor the llama.cpp load log for layers on GPU. - Wrong chat template produces rambling, role confusion, or broken tool calls. Use the
model’s own template (
--jinja, Ollama’s library models). - Concurrency multiplies KV memory:
OLLAMA_NUM_PARALLEL=4at 16K context needs four 16K caches. - Small quantized models fail at tool calling and strict JSON more often than their benchmarks suggest; use grammar/JSON-schema constrained output where the runtime supports it, and validate.
- Thermal throttling on laptops halves sustained throughput; benchmark for minutes, not seconds.
Output contract
**Model:** <name, params, license>, quant <type>, context <n>
**Runtime:** <Ollama | llama.cpp | MLX | vLLM> on <hardware, memory>
**Memory:** estimated <GB> (weights <a>, KV <b>), observed <GB>, all layers on GPU: <yes/no>
**Speed:** prefill <t/s>, decode <t/s>, TTFT <s> at <prompt length>
**Quality:** <score> vs. reference <score> on <n> tasks
**Endpoint:** <URL>, auth <method>, bound to <interface>Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/vram_estimate.py
#!/usr/bin/env python3
"""Estimate memory needed to run a local LLM: weights + KV cache + runtime overhead.
Usage:
python3 vram_estimate.py --params 8 --quant Q4_K_M --ctx 8192 --config config.json [--vram 12]
python3 vram_estimate.py --params 70 --quant Q4_K_M --ctx 16384 --layers 80 --kv-heads 8 --head-dim 128 --vram 48
python3 vram_estimate.py --params 8 --quant Q4_K_M --ctx 32768 --config config.json --kv-type q8_0 --parallel 4
--params model size in billions of parameters
--config a Hugging Face config.json (reads num_hidden_layers, num_key_value_heads, head_dim)
--layers/--kv-heads/--head-dim architecture values if no config.json is available
--kv-type KV cache precision: f16 (default), q8_0, q4_0
--parallel concurrent sequences (each gets its own --ctx worth of KV cache)
--vram available GPU / unified memory in GB; exit 1 if the estimate does not fit
Estimates are approximate (typically within 10-15%). Mixture-of-experts models need memory
for ALL experts' weights even though only some are active per token.
"""
import argparse
import json
import sys
BITS_PER_WEIGHT = { # effective bits incl. scales, approximate for llama.cpp GGUF types
"F32": 32.0, "F16": 16.0, "BF16": 16.0, "Q8_0": 8.5, "Q6_K": 6.6, "Q5_K_M": 5.7,
"Q5_K_S": 5.5, "Q4_K_M": 4.85, "Q4_K_S": 4.6, "Q4_0": 4.55, "IQ4_XS": 4.25,
"Q3_K_M": 3.9, "IQ3_M": 3.7, "Q2_K": 3.35, "AWQ4": 4.25, "GPTQ4": 4.25, "FP8": 8.0,
}
KV_BYTES = {"f16": 2.0, "q8_0": 1.0625, "q4_0": 0.5625}
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--params", type=float, required=True)
ap.add_argument("--quant", default="Q4_K_M", type=str.upper)
ap.add_argument("--ctx", type=int, default=8192)
ap.add_argument("--config")
ap.add_argument("--layers", type=int)
ap.add_argument("--kv-heads", type=int)
ap.add_argument("--head-dim", type=int)
ap.add_argument("--kv-type", default="f16", choices=sorted(KV_BYTES))
ap.add_argument("--parallel", type=int, default=1)
ap.add_argument("--overhead-gb", type=float, default=0.8, help="runtime + compute buffers")
ap.add_argument("--vram", type=float)
a = ap.parse_args()
if a.quant not in BITS_PER_WEIGHT:
print(f"error: unknown quant {a.quant}; known: {', '.join(BITS_PER_WEIGHT)}", file=sys.stderr)
sys.exit(2)
layers, kv_heads, head_dim = a.layers, a.kv_heads, a.head_dim
if a.config:
try:
cfg = json.load(open(a.config, encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
print(f"error: cannot read {a.config}: {exc}", file=sys.stderr)
sys.exit(2)
cfg = cfg.get("text_config", cfg) # multimodal configs nest the LLM
layers = layers or cfg.get("num_hidden_layers")
heads = cfg.get("num_attention_heads")
kv_heads = kv_heads or cfg.get("num_key_value_heads", heads)
head_dim = head_dim or cfg.get("head_dim") or (cfg["hidden_size"] // heads if heads and cfg.get("hidden_size") else None)
if not (layers and kv_heads and head_dim):
print("error: need --config or all of --layers --kv-heads --head-dim for the KV cache estimate", file=sys.stderr)
sys.exit(2)
weights_gb = a.params * 1e9 * BITS_PER_WEIGHT[a.quant] / 8 / 1e9
kv_per_token = 2 * layers * kv_heads * head_dim * KV_BYTES[a.kv_type] # K and V
kv_gb = kv_per_token * a.ctx * a.parallel / 1e9
total = weights_gb + kv_gb + a.overhead_gb
print(f"weights {a.params:g}B @ {a.quant:<7} {weights_gb:7.2f} GB")
print(f"KV cache {a.ctx} ctx x {a.parallel} seq @ {a.kv_type:<5} {kv_gb:7.2f} GB"
f" ({kv_per_token / 1024:.0f} KiB/token; layers={layers} kv_heads={kv_heads} head_dim={head_dim})")
print(f"overhead {a.overhead_gb:7.2f} GB")
print(f"total {total:7.2f} GB")
if a.vram is not None:
usable = a.vram * 0.9 # leave headroom for the OS/display/other processes
if total <= usable:
print(f"FITS in {a.vram:g} GB (using ~90% = {usable:.1f} GB as the usable budget)")
else:
short = total - usable
print(f"DOES NOT FIT in {a.vram:g} GB (short by {short:.1f} GB). Options: smaller quant, "
f"shorter --ctx, --kv-type q8_0, fewer --parallel, or partial GPU offload (slower).")
sys.exit(1)
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.