LLM Observability skill for Claude Code

An LLM feature is observable when any bad answer a user reports can be traced to the exact rendered prompt, model, parameters, retrieved context, tool calls, and response within minutes; when cost, latency, and error rates are on a dashboard per feature; and when user feedback flows into the eval set. Content logging is deliberate: redacted, access-controlled, and expired on a schedule.

Ground rules

• One trace per user request, with child spans for every model call, retrieval, and tool call. Propagate the trace ID to the frontend so feedback links to it.
• Log usage and stop_reason from every model response, always. Content (prompts and outputs) is logged under an explicit policy.
• Redact before export. Never ship raw secrets, credentials, or unnecessary PII to a third-party tracing vendor.
• Record versions on every span: model ID, prompt version, retrieval index version, app release.
• Alerts are on rates and budgets (error rate, p95 latency, spend per hour), not on individual failures.

Workflow

- [ ] 1. Choose the backend
- [ ] 2. Wrap the LLM client with tracing and structured logs
- [ ] 3. Define the content logging policy (redaction, sampling, retention, access)
- [ ] 4. Build dashboards per feature
- [ ] 5. Set alerts
- [ ] 6. Wire feedback to traces and into the eval set

1. Choose the backend

Situation	Choice
Already run an OpenTelemetry backend (Grafana, Honeycomb, Datadog, Jaeger)	Emit OTel spans with GenAI semantic-convention attributes
Need prompt/response inspection UI, eval datasets, prompt versions; open source / self-host	Langfuse or Arize Phoenix
Managed, LangChain-heavy stack	LangSmith
Want zero code changes, accept a proxy hop	A gateway/proxy that logs requests (e.g. Helicone, or your own API gateway)

Pick one system of record for traces. Several half-wired tools are worse than one.

2. Wrap the client

Python, OpenTelemetry plus structured log, around the Anthropic SDK (same shape for any provider):

import hashlib, json, logging, time
from opentelemetry import trace
import anthropic

tracer = trace.get_tracer("llm")
log = logging.getLogger("llm")
client = anthropic.Anthropic(timeout=60, max_retries=3)

PRICES = {"claude-sonnet-5-5": (2.00, 10.00)}  # $/MTok; load from config, verify on pricing page

def traced_create(*, feature: str, prompt_version: str, user_id: str, **params):
    model = params["model"]
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", params.get("max_tokens", 0))
        span.set_attribute("app.feature", feature)
        span.set_attribute("app.prompt_version", prompt_version)
        span.set_attribute("app.user_hash", hashlib.sha256(user_id.encode()).hexdigest()[:16])
        t0 = time.perf_counter()
        try:
            resp = client.messages.create(**params)
        except anthropic.APIStatusError as e:
            span.record_exception(e)
            span.set_attribute("error.type", type(e).__name__)
            log.error(json.dumps({"event": "llm_error", "feature": feature, "model": model,
                                  "status": e.status_code, "error": type(e).__name__}))
            raise
        dt = time.perf_counter() - t0
        u = resp.usage
        cin, cout = PRICES.get(model, (0, 0))
        cache_read = u.cache_read_input_tokens or 0
        cache_write = u.cache_creation_input_tokens or 0
        cost = (u.input_tokens * cin + cache_write * cin * 1.25 + cache_read * cin * 0.1
                + u.output_tokens * cout) / 1e6   # approximate; multipliers vary by model and TTL
        span.set_attribute("gen_ai.response.model", resp.model)
        span.set_attribute("gen_ai.response.id", resp.id)
        span.set_attribute("gen_ai.response.finish_reasons", [resp.stop_reason])
        span.set_attribute("gen_ai.usage.input_tokens", u.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", u.output_tokens)
        span.set_attribute("app.cache_read_tokens", cache_read)
        span.set_attribute("app.cost_usd", cost)
        log.info(json.dumps({
            "event": "llm_call", "feature": feature, "model": model, "prompt_version": prompt_version,
            "latency_s": round(dt, 3), "input_tokens": u.input_tokens, "output_tokens": u.output_tokens,
            "cache_read": cache_read, "cache_write": cache_write, "stop_reason": resp.stop_reason,
            "cost_usd": round(cost, 6), "request_id": getattr(resp, "_request_id", None),
            "trace_id": format(span.get_span_context().trace_id, "032x"),
        }))
        return resp

• The OTel GenAI semantic conventions are still evolving (attribute names such as gen_ai.system were renamed to gen_ai.provider.name); check the version your backend expects and keep custom fields under an app. prefix.
• For streaming, record time to first token as a span event or attribute when the first text delta arrives, and usage from the final message.
• For agents, create a parent span per run and child spans per turn, tool call (execute_tool <name> with arguments redacted), and retrieval (query, top-k IDs, scores, index version).
• Capture the provider’s request ID; support tickets with the provider need it.
• Some SDKs and instrumentation libraries emit GenAI spans automatically; prefer them when they exist, and add the app-level attributes above.

3. Content logging policy

Decision	Default
What content to log	Rendered prompts and outputs for a sampled fraction (e.g. 5-10%) plus 100% of errors, negative feedback, and flagged outputs
Redaction	Before export: secrets (API keys, tokens), emails, phone numbers, card numbers, government IDs via regex/NER; replace with typed placeholders
User identity	Hashed or pseudonymous IDs; mapping table kept separately with stricter access
Retention	Content 14-30 days; metrics and usage indefinitely; honor deletion requests
Access	Role-restricted; access to content is audited
Opt-outs	Respect customer contracts and zero-retention agreements: some tenants get metadata-only logging

Document the policy in the repo and enforce it in the wrapper, not in each call site.

4. Dashboards

Per feature and model:

Panel	Why
Requests, error rate by error type (429, 5xx, timeout, refusal)	Availability and provider health
Latency p50/p95/p99 and TTFT p95	User experience
Input, output, and cached tokens per request	Context bloat and caching health
Cost per hour/day and cost per request	Budget and regressions
Cache hit ratio (cache_read / total input on cacheable traffic)	Detects silent cache invalidation
stop_reason distribution	Rising max_tokens = truncation; rising refusal = prompt or content shift
Agent turns and tool calls per run	Loops and runaway agents
Feedback rate (thumbs down / total) by prompt version	Quality signal tied to releases

5. Alerts

• Error rate > 2-5% over 10 minutes (page); provider 429s sustained (warn).
• p95 latency above budget for 15 minutes.
• Spend per hour > 2x trailing 7-day average for that hour (catches loops and abuse).
• max_tokens stop share above baseline after a release.
• Cache hit ratio drops by more than half after a deploy.

6. Feedback loop

• Return the trace ID with each response; the thumbs-up/down endpoint stores {trace_id, rating, comment} and annotates the trace.
• Weekly: review negative-feedback traces, tag failure modes, and add representative ones (redacted) to the eval set (llm-evals).
• Compare feedback rate and eval scores per prompt version after each release.

Verification

1. Make a request in staging and find its full trace by trace ID: rendered prompt (redacted), model, tokens, cost, tool spans, and final output are all present.
2. Send a synthetic secret (sk-test-CANARY) and an email address through the app; confirm both are redacted in the exported trace.
3. Force a 429 and a timeout with mocks; confirm error spans, logs, and dashboard counts.
4. Compare the dashboard’s daily cost with the provider’s usage report; investigate any gap above ~5%.

Gotchas

• Logging only the user’s message, not the fully rendered prompt, makes template and retrieval bugs invisible.
• High-cardinality attributes (full prompts as metric labels, raw user IDs) blow up metrics backends; keep them on spans and logs, not metric dimensions.
• Streaming responses closed early by the client never log usage unless you handle cancellation; record partial usage on disconnect.
• Third-party tracing SDKs that auto-capture content may send PII to a vendor by default; check the capture settings before enabling in production.
• Cost computed from list prices drifts from the invoice (discounts, batch, cache TTL pricing); reconcile against the provider’s usage data.

Output contract

**Backend:** <OTel -> X | Langfuse | Phoenix | ...>; wrapper <path>
**Captured per call:** model, versions, tokens, cache, cost, latency/TTFT, stop_reason, request ID
**Content policy:** sample <x%> + errors/feedback; redaction <rules>; retention <days>; access <roles>
**Dashboards/alerts:** <links>; alert thresholds <list>
**Feedback loop:** trace-linked ratings -> weekly triage -> eval set