LLM Gateway Routing skill for Claude Code

Routing is done well when every route has a written contract (quality bar, latency budget, cost ceiling), every provider error maps to exactly one action, every fallback model has passed the route’s eval, retries are bounded by a deadline and a budget, and a per-route report shows error rate, fallback rate, p95 latency and cost per request. A fallback list nobody has tested is not resilience; it is a second, unknown product.

Ground rules

• Retry at one layer only. If a gateway retries, set the provider SDK to max_retries=0 (Anthropic and OpenAI SDKs retry twice by default) or attempts multiply.
• A fallback model must pass the same eval as the primary on that route’s test set before it goes in the config. Prompts tuned for one model often degrade on another.
• Never fall back on a deterministic client error (400, 401, 403, 404, 413). It fails the same way everywhere, and fallback hides the bug.
• Streaming: fall back only before the first token reaches the user. After that, surface an error or restart visibly; never splice two models’ output.
• Every attempt is logged with route, model, status, latency, tokens and cost. Without per-attempt logs you cannot see retry amplification or silent fallback.

Do you need a gateway?

Situation	Recommendation
One provider, one or two features, low volume	No gateway. Official SDK in one client module (see llm-app-builder) with SDK retries and a timeout
One provider, need a second for outages	In-app router (below), ~100 lines, no new infrastructure
Several services or teams share keys, need per-team budgets, virtual keys, central logs	Self-hosted proxy (LiteLLM) or a managed gateway
No ops capacity, want fallbacks plus analytics fast	Managed gateway (OpenRouter, Cloudflare AI Gateway, Vercel AI Gateway, Portkey); check data retention and zero-data-retention terms first
Strict data residency or regulated data	Self-hosted proxy in your network, or direct SDK calls; a third-party gateway is another processor in the data path

A gateway adds a network hop, a single point of failure, and a lowest-common-denominator API. If you rely on provider-specific features (prompt caching breakpoints, extended tool types, structured output modes, citations), verify the gateway passes them through unchanged before adopting it.

Workflow

- [ ] 1. Inventory routes: feature x task, current model, volume, p95, error rate, cost
- [ ] 2. Write a contract per route (quality bar, latency budget, cost ceiling, data constraints)
- [ ] 3. Build the error-to-action table
- [ ] 4. Choose tiering: static per route first, difficulty-based only with an eval
- [ ] 5. Set deadlines: TTFT timeout, total timeout, per-attempt timeout
- [ ] 6. Set the retry budget and cooldowns
- [ ] 7. Implement (in-app, LiteLLM, or managed) and log every attempt
- [ ] 8. Prove it with fault injection and the route report

1-2. Inventory and contracts

Grep for every model call (messages.create, chat.completions.create, generateText, litellm.completion) and list them as routes. For each, record in a table:

| Route | Primary | Fallback | Quality bar | TTFT budget | Total budget | Cost ceiling/req | Data constraint |
|---|---|---|---|---|---|---|---|
| support-chat | sonnet-tier | other-provider mid-tier | eval >= 0.85 | 1.5 s | 20 s | $0.02 | EU only |
| ticket-tagging | haiku-tier | none (queue + retry later) | macro-F1 >= 0.9 | n/a | 10 s | $0.001 | none |

Background routes often need no fallback at all: a queue with delayed retry is cheaper and keeps one model’s behavior.

3. Error-to-action table

Signal	Action
429 with retry-after	If retry-after fits in the remaining deadline: wait and retry same model. Otherwise fall back
429 without retry-after and a spend-cap code (Anthropic error.details.error_code: enforced_spend_limit_reached)	Do not retry; it fails until the cap resets. Fall back or fail fast and alert
529 overloaded_error (Anthropic), 503	Retry once with jittered backoff, then fall back
500 / 502 / 504	Retry once, then fall back
Connection reset, DNS, TLS errors	Retry once, then fall back
Per-attempt timeout (no first token within TTFT budget)	Cancel the request, fall back; do not wait for the slow one
400 context length exceeded	Route to a larger-context model or trim context; never retry as-is
400 / 401 / 403 / 404 / 413 other	Fail, log, alert. No retry, no fallback
Refusal or content-policy stop	Treat as a product decision: show the refusal path, or use an explicit policy fallback route that is reviewed, never a silent provider swap
Mid-stream error after tokens were sent	Fail visibly with a retry button; log as partial

Read rate-limit headers to slow down before you hit 429: Anthropic returns anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining, anthropic-ratelimit-output-tokens-remaining and matching -reset timestamps; OpenAI returns x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens.

4. Tiering by difficulty

Default: one model per route, chosen by eval. Add dynamic tiering only when a route has both cheap and hard traffic and volume makes the saving matter.

Pattern	How	When
Static per route	Config maps route -> model	Default
Rule-based	Length, attachment count, user plan, keyword -> tier	Signals are obvious and cheap
Cascade	Cheap model answers with a self-check or validator; escalate on failure	Output is verifiable (schema, tests, citations)
Classifier router	Small model or embedding classifier picks the tier	High volume, labeled examples of hard vs easy

Measure a cascade on the eval set: escalation rate, quality of the cheap tier on the traffic it keeps, and total cost including the wasted cheap calls. If more than about half of traffic escalates, the cascade costs more than calling the strong model directly.

5. Deadlines

• Set two timeouts for streaming routes: time to first token (the gateway or client aborts if no chunk arrives) and total time. LiteLLM calls these stream_timeout and timeout per deployment.
• Derive per-attempt timeouts from the user-facing budget: attempts x per_attempt_timeout + backoff <= total budget. A 30 s budget with a 60 s SDK timeout means the fallback never runs.
• Propagate the remaining deadline into each attempt instead of restarting the clock.

6. Retry budget and cooldowns

• Per request: at most 2 attempts on the same model plus one fallback, all inside the deadline.
• Global: cap retries at a fixed share of traffic (10% is a common starting point) so a provider incident does not double your load on it. Reject retries once the budget is spent.
• Cooldown: after N failures within a window, take a deployment out of rotation for a period (LiteLLM allowed_fails and cooldown_time). Add jitter so all workers do not return at once.
• Long generations that time out are expensive to retry; prefer streaming and a TTFT timeout so failure is detected in seconds, not minutes.

For general retry and circuit-breaker mechanics beyond LLM calls, see resilience-patterns.

7. Implement

In-app router (Python, provider-agnostic sketch):

import random, time

RETRYABLE = {408, 429, 500, 502, 503, 504, 529}

class RetryBudget:
    def __init__(self, ratio=0.1, floor=10):
        self.requests = 0; self.retries = 0; self.ratio = ratio; self.floor = floor
    def allow(self):
        return self.retries < max(self.floor, self.ratio * self.requests)

budget = RetryBudget()

def call_route(route, payload, deadline_s):
    start = time.monotonic()
    budget.requests += 1
    for i, target in enumerate(route.targets):          # primary first, then fallbacks
        for attempt in range(route.max_attempts_per_target):
            remaining = deadline_s - (time.monotonic() - start)
            if remaining <= 0.5:
                raise TimeoutError(f"{route.name}: deadline exhausted")
            if (i or attempt) and not budget.allow():
                raise RuntimeError(f"{route.name}: retry budget exhausted")
            if i or attempt:
                budget.retries += 1
            try:
                result = target.call(payload, timeout=min(target.timeout_s, remaining))
                log_attempt(route.name, target.model, "ok", result.usage, start)
                return result
            except ProviderError as e:                   # your normalized error type
                log_attempt(route.name, target.model, e.status, None, start)
                if e.spend_cap or e.status not in RETRYABLE:
                    if e.status in (400, 401, 403, 404, 413) and not e.context_length:
                        raise                               # deterministic: no fallback
                    break                                   # skip to next target
                wait = e.retry_after or random.uniform(0, 2 ** attempt)
                if wait > remaining - 1:
                    break                                   # fall back instead of waiting
                time.sleep(wait)
    raise RuntimeError(f"{route.name}: all targets failed")

LiteLLM proxy (config.yaml): keys verified against the LiteLLM docs; model IDs are placeholders to fill from each provider’s current model list.

model_list:
  - model_name: chat-primary            # alias your app calls
    litellm_params:
      model: anthropic/<model-id>
      api_key: os.environ/ANTHROPIC_API_KEY
      timeout: 30                       # whole response, seconds
      stream_timeout: 3                 # first chunk, seconds
  - model_name: chat-fallback
    litellm_params:
      model: openai/<model-id>
      api_key: os.environ/OPENAI_API_KEY
      timeout: 30
      stream_timeout: 3

router_settings:
  routing_strategy: simple-shuffle      # default; spreads load across same-alias deployments
  num_retries: 1
  allowed_fails: 3
  cooldown_time: 30
  enable_pre_call_checks: true          # needed for context-window checks
  fallbacks: [{"chat-primary": ["chat-fallback"]}]
  context_window_fallbacks: [{"chat-primary": ["chat-fallback"]}]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Run with litellm --config config.yaml (or the official Docker image) and point clients at it with the OpenAI-compatible SDK. retry_policy can set retry counts per exception type (RateLimitErrorRetries, TimeoutErrorRetries, and others). Set per-key or per-team budgets through virtual keys when several teams share the proxy.

Managed gateways: OpenRouter accepts a models array in priority order and bills the model that actually served (read the response model field). Cloudflare AI Gateway supports per-request timeouts, retries and fallbacks. For any managed gateway, confirm in its current docs: which errors trigger fallback, whether logs store prompts, and how provider keys are held.

8. Prove it

1. Fault injection in staging: point the primary at a stub returning 429 (with and without retry-after), 529, 500, a 10 s stall before first byte, and a mid-stream disconnect. Confirm each produces the action in the table, inside the deadline.
2. Run the eval on every fallback target for every route that uses it.
3. Run the route report on a day of attempt logs:

python3 scripts/route_report.py attempts.jsonl --prices prices.json \
  --max-error-rate 0.01 --max-p95-ms 4000

It prints per route: final error rate, retry and fallback rates, attempts per request (retry amplification), p50/p95 end-to-end latency, p95 TTFT, cost per request, and which models served traffic. Exit code 1 means a route breached a threshold.

Gotchas

• Prompt caching is per provider and usually per model. Failing over moves traffic onto a cold cache, so the fallback can cost several times more per request; budget for it.
• Tokenizers and context windows differ. A prompt that fits the primary can 400 on the fallback; run pre-call token checks or trim per target.
• Tool schemas, JSON modes and stop reasons differ across providers. Normalize them in the router and test tool-calling routes against each target.
• “OpenAI-compatible” endpoints accept the request shape but may drop provider-only parameters silently. Diff the responses, not just the status codes.
• Fallback onto a provider without your data-processing or zero-retention agreement is a compliance incident. Encode allowed providers per route.
• A fallback that serves 30% of traffic for a week is a migration nobody approved. Alert when the fallback rate exceeds a few percent for more than an hour.
• Latency-based routing chases noise at low volume; keep simple-shuffle until each deployment sees steady traffic.

Output contract

## Routing table
| Route | Primary | Fallbacks (eval score) | TTFT / total budget | Retry policy | Cost ceiling |
## Error-to-action table
## Implementation: in-app | LiteLLM | managed (why)
## Fault-injection results (scenario -> observed action -> within deadline?)
## Route report (scripts/route_report.py output)