LLM App Builder skill for Claude Code

An LLM feature is production-ready when every call has a timeout, typed error handling, a known per-request cost, a streaming path for anything a human waits on, and an eval that proves the chosen model is good enough. “It works in the playground” is not ready.

Ground rules

• Put every model call behind one module (llm/client.ts, llm/client.py). Feature code never imports the SDK directly. This is where retries, logging, and model IDs live.
• Model IDs come from config, never string literals scattered through the code.
• Use the official SDK for the provider. Do not hand-roll HTTP or bolt on an OpenAI-compatibility shim for a provider that has its own SDK.
• Stream any response a user waits on. Non-streamed calls are for background jobs and short classifications.
• Log usage from every response. You cannot control cost you do not measure.
• Never put secrets, API keys, or raw user PII into prompts or logs without a reason.

Workflow

- [ ] 1. Define the task contract (input, output shape, latency and cost budget)
- [ ] 2. Pick the tier: single call, workflow, or agent
- [ ] 3. Choose the model with a small eval, not by vibes
- [ ] 4. Build the client module (timeouts, retries, typed errors)
- [ ] 5. Add streaming for user-facing paths
- [ ] 6. Control tokens: max_tokens, context trimming, caching
- [ ] 7. Instrument usage, latency, and errors
- [ ] 8. Verify with an eval set and a load smoke test

1. Define the task contract

Write down before coding:

• Input: what the model receives and its maximum size in tokens.
• Output: free text, or a schema (then use structured-outputs).
• Latency budget: time-to-first-token (TTFT) for chat, total time for jobs.
• Cost budget per request and per day.
• Failure behavior: what the user sees if the provider is down.

2. Pick the tier

Task shape	Tier	Notes
Classify, summarize, extract, answer from given context	Single call	Most features belong here
Fixed multi-step pipeline (extract, then validate, then draft)	Workflow: code calls the model several times	Deterministic control flow, easy to test
Open-ended steps the model must choose	Agent loop with tools	See tool-use-agents; costs 5-50x more

Default to the simplest tier that passes the eval.

3. Choose the model

Current Anthropic model IDs (verify against the provider’s models list before shipping):

Model ID	Use for
claude-haiku-5-5	High-volume classification, routing, extraction, cheap subtasks
claude-sonnet-5-5	Everyday chat, coding help, most product features
claude-opus-5-5	Hard reasoning, agents, long documents, quality-critical output
claude-fable-5-1	The most demanding long-horizon work where cost is secondary

Procedure: run 30-100 representative inputs through two candidate models, grade with llm-evals, and pick the cheapest model that meets the bar. Tune reasoning effort (output_config: {"effort": "low" | "medium" | "high"} on current Claude models) before jumping a model tier; lower effort on a stronger model often beats a weaker model.

4. Build the client module

Python:

import anthropic
from anthropic import Anthropic

client = Anthropic(timeout=60.0, max_retries=3)  # retries 408/409/429/5xx with backoff

def complete(system: str, user: str, *, model: str, max_tokens: int = 2048) -> str:
    try:
        resp = client.messages.create(
            model=model,
            max_tokens=max_tokens,
            system=system,
            messages=[{"role": "user", "content": user}],
        )
    except anthropic.RateLimitError:
        raise            # already retried; surface as 503 + Retry-After upstream
    except anthropic.APIConnectionError:
        raise            # network; already retried
    except anthropic.BadRequestError as e:
        raise ValueError(f"bad request, do not retry: {e}") from e
    if resp.stop_reason == "max_tokens":
        log.warning("truncated output", extra={"model": model})
    if resp.stop_reason == "refusal":
        return FALLBACK_MESSAGE
    return "".join(b.text for b in resp.content if b.type == "text")

TypeScript:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ timeout: 60_000, maxRetries: 3 }); // timeout is ms in TS

export async function complete(system: string, user: string, model: string) {
  const resp = await client.messages.create({
    model,
    max_tokens: 2048,
    system,
    messages: [{ role: "user", content: user }],
  });
  return resp.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
}

Rules for the module:

• Timeout units differ: Python seconds, TypeScript milliseconds. Wall-clock worst case is timeout x (max_retries + 1); size it against the user-facing budget.
• Catch the SDK’s typed exceptions most-specific-first. Never string-match error text.
• Retry only 408/409/429/5xx and connection errors. A 400 will fail identically forever.
• Respect retry-after on 429. The SDK does this; custom retry loops often do not.
• Always check stop_reason: end_turn (done), max_tokens (truncated), tool_use, refusal (show a fallback, do not parse content as normal output).

5. Stream user-facing responses

Python (server-sent events from FastAPI):

from fastapi.responses import StreamingResponse

async def chat_stream(messages, model):
    async with async_client.messages.stream(
        model=model, max_tokens=4096, messages=messages
    ) as stream:
        async for text in stream.text_stream:
            yield f"data: {json.dumps({'delta': text})}\n\n"
        final = await stream.get_final_message()
        yield f"data: {json.dumps({'done': True, 'usage': final.usage.model_dump()})}\n\n"

@app.post("/chat")
async def chat(req: ChatRequest):
    return StreamingResponse(chat_stream(req.messages, MODEL), media_type="text/event-stream")

TypeScript:

const stream = client.messages.stream({ model, max_tokens: 4096, messages });
stream.on("text", (delta) => res.write(`data: ${JSON.stringify({ delta })}\n\n`));
const final = await stream.finalMessage();
res.end(`data: ${JSON.stringify({ done: true, usage: final.usage })}\n\n`);

Streaming gotchas:

• Errors can arrive mid-stream after HTTP 200. Send an explicit error event to the client.
• Disable proxy buffering (X-Accel-Buffering: no for nginx) or tokens arrive in one lump.
• Abort the upstream request when the browser disconnects, or you pay for unread tokens.
• Large max_tokens (tens of thousands) requires streaming to avoid HTTP timeouts.

6. Control tokens

Lever	Action
max_tokens	Size to the real output: ~256 for labels, 2-4K for answers, higher only when streaming
History	Keep the last N turns plus a running summary; never send unbounded chat history
Retrieved context	Send top-k chunks, not whole documents (see rag-pipelines)
Repeated prefix	Stable system prompt + tools first, then mark with cache_control: {"type": "ephemeral"}
Offline work	Use the provider’s batch API (about half price on Anthropic)
Counting	Use the provider’s token-count endpoint (client.messages.count_tokens); never estimate with another vendor’s tokenizer

For deeper cost work, use prompt-caching-costs.

7. Instrument

Log per call: model, latency, TTFT (streaming), input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, stop_reason, request ID, feature name, and a hashed user ID. Details in llm-observability.

8. Verify

1. Run the eval set against the configured model; pass rate meets the agreed bar.
2. Force each failure path in tests: mock a 429, a 500, a timeout, a max_tokens stop, and a refusal stop. Each produces the intended user-facing behavior.
3. Smoke-load the endpoint at expected peak concurrency; confirm no 429 storm and p95 latency within budget.
4. Compute cost per request from logged usage and compare to the budget.

Hard rules and gotchas

• Current Claude models reject assistant-message prefill and non-default temperature values on several models. Steer format with instructions or structured outputs, and do not count on temperature=0 for determinism on any provider.
• The API is stateless: you resend history every turn. Conversation cost grows quadratically with turns unless you trim and cache.
• Parse tool and JSON outputs with a JSON parser, never regex on serialized text.
• Keep provider specifics (model IDs, parameter names) inside the client module so a provider switch touches one file. The architecture above applies to any provider; only the SDK calls change.
• Do not call the model in a request path that has no timeout; an upstream stall will exhaust your worker pool.
• Rate limits are per organization and per model. A batch backfill sharing the production key can starve live traffic; give offline jobs their own key or workspace.

Output contract

When adding an LLM feature, deliver:

**Feature:** <name> — <tier: single call | workflow | agent>
**Model:** <id> (effort: <level>) — chosen because <eval result vs. alternative>
**Budgets:** TTFT <x> s p95, cost <$y> per request, max input <n> tokens
**Failure behavior:** 429 -> <...>; timeout -> <...>; refusal -> <...>
**Files:** llm/client.<ext>, <feature files>, tests for each failure path
**Eval:** <n> cases, pass rate <p>% (command: <...>)