OpenTelemetry skill for Claude Code

Instrumentation is done when one user request produces a single connected trace across every service it touches, spans follow semantic conventions, metrics have bounded label cardinality, logs carry trace_id, and telemetry flows through a Collector you can reconfigure without redeploying apps.

Ground rules

• Start with auto-instrumentation; add manual spans only around business operations the auto layer cannot see.
• Initialize the SDK before any instrumented library is imported.
• Set service.name, service.version, and deployment.environment.name on every service.
• Send to a Collector (OTLP), not directly to vendors, so sampling, scrubbing, and routing live in one place.
• Never put PII, secrets, or unbounded values (user IDs, raw URLs with IDs) into metric attributes. Be careful with span attributes too.
• Flush on shutdown; short-lived processes and serverless lose buffered spans otherwise.

Workflow

- [ ] 1. Inventory services, languages, existing telemetry, backend
- [ ] 2. Stand up a Collector (or local Jaeger) to see output immediately
- [ ] 3. Enable auto-instrumentation with env config
- [ ] 4. Add manual spans/metrics for key business operations
- [ ] 5. Propagate context across async boundaries (queues, jobs)
- [ ] 6. Correlate logs with trace IDs
- [ ] 7. Configure sampling and processors; verify end-to-end traces

2. Local backend for verification

docker run --rm -p 16686:16686 -p 4317:4317 -p 4318:4318 jaegertracing/jaeger:latest
# UI: http://localhost:16686 ; OTLP gRPC :4317, OTLP HTTP :4318

Pin the Jaeger tag in anything committed.

3. Auto-instrumentation

Standard env vars work across SDKs:

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.14.2,deployment.environment.name=staging
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=1.0          # 100% in dev; lower in prod

Language	Zero-code setup
Node.js	npm i @opentelemetry/api @opentelemetry/auto-instrumentations-node then node --require @opentelemetry/auto-instrumentations-node/register app.js
Python	pip install opentelemetry-distro opentelemetry-exporter-otlp && opentelemetry-bootstrap -a install then opentelemetry-instrument python app.py
Java	java -javaagent:opentelemetry-javaagent.jar -jar app.jar
.NET	OpenTelemetry .NET automatic instrumentation, or the OpenTelemetry.Extensions.Hosting packages
Go	No agent by default: SDK in code plus contrib wrappers (otelhttp, otelgrpc, otelsql)

Node SDK in code (when you need custom config; SDK 2.x API):

// instrumentation.mjs — load first: node --import ./instrumentation.mjs app.mjs
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'checkout',
    [ATTR_SERVICE_VERSION]: process.env.APP_VERSION ?? 'dev',
  }),
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() }),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fs': { enabled: false },   // very noisy
  })],
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

ESM apps may need the instrumentation loader hook for some libraries to be patched; check the current @opentelemetry/instrumentation docs for the ESM registration flag.

4. Manual spans and metrics

import { trace, metrics, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('checkout');
const meter = metrics.getMeter('checkout');
const ordersPlaced = meter.createCounter('orders.placed', { unit: '{order}' });
const paymentDuration = meter.createHistogram('payment.duration', { unit: 's' });

export async function placeOrder(cart) {
  return tracer.startActiveSpan('placeOrder', async (span) => {
    span.setAttribute('order.item_count', cart.items.length);
    const start = performance.now();
    try {
      const result = await chargeCard(cart);
      ordersPlaced.add(1, { 'payment.method': cart.method });   // bounded values only
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      paymentDuration.record((performance.now() - start) / 1000, { 'payment.method': cart.method });
      span.end();
    }
  });
}

from opentelemetry import trace
tracer = trace.get_tracer("checkout")

def place_order(cart):
    with tracer.start_as_current_span("place_order") as span:
        span.set_attribute("order.item_count", len(cart.items))
        return charge_card(cart)   # exceptions are recorded and status set automatically

Span naming: low-cardinality operation names (GET /orders/{id}, placeOrder), never GET /orders/12345. Put IDs in attributes, not names.

Instrument choice: counter for things that only go up; up-down counter for in-flight counts; histogram for durations and sizes (unit seconds); gauge (observable) for sampled values like queue depth.

5. Context propagation

• HTTP: auto-instrumentation injects/extracts W3C traceparent/tracestate. If traces break at a hop, check that proxies and API gateways forward these headers.
• Queues (Kafka, SQS, RabbitMQ): inject into message headers on produce, extract on consume. Use the library’s instrumentation when available; otherwise:

import { context, propagation, trace } from '@opentelemetry/api';
const headers = {};
propagation.inject(context.active(), headers);       // producer: attach headers to message
const parent = propagation.extract(context.active(), msg.headers);   // consumer
await context.with(parent, () => tracer.startActiveSpan('process order', async (s) => { /* ... */ s.end(); }));

• Batch consumers: create one span per batch with span links to each message’s context rather than pretending one parent.

6. Logs

Inject trace_id and span_id into structured logs (pino, winston, structlog, logback have OTel integrations or MDC support), so a log line links to its trace. Ship logs via the Collector (filelog receiver) or the SDK log bridge, not both.

7. Collector pipeline

# otel-collector.yaml (otelcol-contrib)
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }
  attributes/scrub:
    actions:
      - { key: http.request.header.authorization, action: delete }
      - { key: user.email, action: delete }
  batch: {}
exporters:
  otlphttp/backend:
    endpoint: https://otlp.example-vendor.com
    headers: { authorization: "Bearer ${env:OTLP_TOKEN}" }
  debug: { verbosity: basic }
extensions:
  health_check: {}
service:
  extensions: [health_check]
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, attributes/scrub, batch], exporters: [otlphttp/backend] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/backend] }
    logs:    { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/backend] }

memory_limiter goes first, batch last. Validate: otelcol-contrib validate --config otel-collector.yaml.

Sampling:

Volume	Strategy
Low (< ~100 req/s total)	Keep 100%
Medium/high	Head sampling parentbased_traceidratio at 5–20% in SDKs
Need all errors and slow traces	Tail sampling processor in the Collector; route all spans of a trace to the same Collector (loadbalancing exporter by trace ID)

Always use a parent-based sampler so downstream services honor the upstream decision.

Verification

1. Send one request; in Jaeger/your backend, find a single trace containing spans from every service in the path, with correct parent-child nesting.
2. Check span attributes use semantic convention names (http.request.method, http.response.status_code, url.path, db.system.name or the convention version your SDK emits).
3. Query metrics: series count per metric stays small (no IDs in attributes).
4. Pick a log line from the request; its trace_id opens the same trace.
5. Kill the process with SIGTERM during load; spans up to shutdown still arrive.

Gotchas

• Port/protocol mismatch: 4317 is gRPC, 4318 is HTTP. With the generic OTEL_EXPORTER_OTLP_ENDPOINT, HTTP exporters append /v1/traces; with the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, you must include the path.
• Importing express/pg before the SDK starts leaves them unpatched: no spans, no error.
• Two instrumentation layers (vendor agent + OTel) produce duplicate spans.
• Semantic convention names changed across versions (http.method -> http.request.method); dashboards built on old names silently go empty after an SDK upgrade.
• Lambda/serverless: call forceFlush() before the handler returns, or use the OTel Lambda layer.
• fs and DNS instrumentation can generate thousands of spans per request; disable them.

Output

**Services instrumented:** checkout (Node, auto + 3 manual spans), payments (Python, auto)
**Resource attributes:** service.name, service.version, deployment.environment.name
**Propagation:** HTTP (W3C), Kafka headers on orders topic
**Collector:** memory_limiter -> scrub -> batch -> vendor OTLP; config validated
**Sampling:** parentbased_traceidratio 0.1 in prod; tail sampling keeps errors
**Proof:** trace 4bf92f3577b34da6a3ce929d0e0e4736 spans checkout -> payments -> Kafka consumer