DevOps & SRE No. 166Editor's pick

Prometheus Alerting skill for Claude Code

Pages that mean act now, routed to one thread. Claude sorts signals into page, ticket or dashboard before writing a single rule, then builds recording rules, symptom-based alerts with for durations, minimum-traffic guards, severity labels and runbook links. Rules are unit tested with promtool and checked by a bundled linter, and Alertmanager gets grouping and inhibition so one outage sends one notification.

nameprometheus-alerting
size249 lines · ≈2,500 tokens
files2
licenseMIT
updatedOct 20, 2026

What Claude sees before loading it

This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.

description: Writes Prometheus metrics queries, recording rules and alerting rules that page only on user-visible symptoms - PromQL for rates, ratios and histogram quantiles, label hygiene and cardinality, for and keep_firing_for durations, severity labels, runbook annotations, Alertmanager routing, grouping, inhibition and silences - and tests them with promtool and amtool. Use when writing or reviewing *.rules.yml, alertmanager.yml or PrometheusRule CRDs, when on-call gets noisy or flapping alerts, or when an outage happened without an alert.

Try asking

Our on-call gets 40 alerts a night. Audit these Prometheus rules and cut the noise
Write alert rules for API error ratio and latency with promtool tests
Set up Alertmanager so critical alerts page PagerDuty and warnings become tickets

Use it when

Writing or reviewing Prometheus rule files or PrometheusRule CRDs
Alerts are noisy, flapping, or missed a real outage
Configuring Alertmanager routes, grouping, inhibition or silences
PromQL for rates, ratios or histogram percentiles returns odd numbers

Skip it when

You are defining SLOs and burn-rate alerts — use slo-error-budgets
You need dashboards rather than alerts — use grafana-dashboards

How to check the result

promtool check rules and promtool test rules pass, scripts/alert_lint.py reports no errors, amtool routes test sends each severity to the intended receiver, and a test alert arrives end to end.

The common mistake it prevents

Paging on causes such as high CPU or a restarted pod instead of user-visible symptoms, which trains on-call to ignore alerts.

~/.claude/skills/prometheus-alerting/SKILL.md

--- name: prometheus-alerting description: Writes Prometheus metrics queries, recording rules and alerting rules that page only on user-visible symptoms - PromQL for rates, ratios… license: MIT ---

An alert set is good when every page means a human must act now on a user-visible problem, every alert has a runbook link and a clear summary, nothing flaps, rules pass promtool test rules, and Alertmanager routes, groups and inhibits so one incident produces one notification thread.

Ground rules

Page on symptoms (error ratio, latency, saturation that will cause user impact soon), not causes (CPU high, pod restarted). Causes become tickets or dashboard panels.
Every alerting rule has: for (or a deliberate reason for none), a severity label, summary and description annotations, and runbook_url.
Use rate() over counters, never raw counter values. Range window at least 4x the scrape interval (with 30 s scrapes, [2m] minimum; [5m] default).
Precompute expensive or reused expressions as recording rules named level:metric:operations (e.g. job:http_requests:rate5m).
No unbounded labels (user ID, full URL, request ID) on metrics. Cardinality kills Prometheus before bad alerts do.
For SLO burn-rate alerting, use slo-error-budgets; this skill covers the rest of the rule set and Alertmanager.

Workflow

- [ ] 1. Inventory current rules, alert history, and noise
- [ ] 2. Decide what deserves a page vs ticket vs dashboard
- [ ] 3. Write recording rules for the SLIs and heavy queries
- [ ] 4. Write alert rules with for, labels, annotations
- [ ] 5. Unit test with promtool; lint with the bundled script
- [ ] 6. Configure Alertmanager routing, grouping, inhibition
- [ ] 7. Deploy, fire a test alert end to end, review noise after a week

1. Inventory

promtool check rules rules/*.yml
curl -s localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.type=="alerting") | {name, state, health}'
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[] | {labels, state, activeAt}'

Find noisy alerts from history (if Prometheus scrapes itself):

topk(10, sum by (alertname) (changes(ALERTS{alertstate="firing"}[7d])))

Card each alert: fires how often, how often acted on, median time firing. Delete or downgrade any alert acted on < 50% of the time.

2. Page, ticket, or dashboard

Condition	Route
Users are failing now (error ratio, latency, availability probe)	Page (`severity: critical`)
Will fail within hours (disk full in < 4 h, cert expiring < 7 d, queue backlog growing)	Page or business-hours ticket by lead time
Degradation without user impact (one replica down of many, high CPU with headroom)	Ticket (`severity: warning`)
Useful context only	Dashboard panel, no alert
Monitoring itself broken (target down, no data)	Ticket; page only for the main Prometheus/Alertmanager

3. PromQL patterns and recording rules

groups:
  - name: http-recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
      - record: job:http_error_ratio:rate5m
        expr: job:http_errors:rate5m / job:http_requests:rate5m
      - record: job:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Need	PromQL
Per-second rate of a counter	`rate(x_total[5m])`; `irate` only for graphs, never alerts
Ratio	aggregate numerator and denominator separately, then divide
Quantile from classic histogram	`histogram_quantile(0.99, sum by (le, job) (rate(x_bucket[5m])))`
Quantile from native histogram	`histogram_quantile(0.99, sum by (job) (rate(x[5m])))`
Predict exhaustion	`predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0`
Metric absent (target vanished)	`absent(up{job="api"})` or `absent_over_time(x[10m])`
Avoid division by zero noise	`... and job:http_requests:rate5m > 1` (minimum traffic guard)

Never average percentiles across instances; aggregate buckets first.

4. Alert rules

groups:
  - name: checkout-alerts
    rules:
      - alert: CheckoutHighErrorRatio
        expr: |
          job:http_error_ratio:rate5m{job="checkout"} > 0.02
            and job:http_requests:rate5m{job="checkout"} > 1
        for: 5m
        keep_firing_for: 5m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: "Checkout 5xx ratio {{ $value | humanizePercentage }} (> 2%)"
          description: "More than 2% of checkout requests failed for 5 minutes."
          runbook_url: "https://runbooks.acme.dev/checkout/high-error-ratio"
          dashboard: "https://grafana.acme.dev/d/checkout"

      - alert: DiskWillFillIn4Hours
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0
            and node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.2
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} fills within 4h"
          runbook_url: "https://runbooks.acme.dev/node/disk-filling"

Rules for for and windows:

Signal	for	Notes
Error ratio / latency page	5m (2-10m)	Pair with min-traffic guard
Saturation prediction	15-30m	predict_linear is noisy on short windows
Target down	5m	Combine with `absent()` for whole-job disappearance
Certificate expiry	1h	Threshold in days, not seconds remaining checks every scrape

keep_firing_for stops resolve/re-fire flapping when the expression dips briefly.

5. Test and lint

promtool test rules with a unit test file:

# tests/checkout_test.yml
rule_files: [../rules/checkout.yml, ../rules/http-recording.yml]
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+100x30'
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+5x30'
    alert_rule_test:
      - eval_time: 12m
        alertname: CheckoutHighErrorRatio
        exp_alerts:
          - exp_labels: { severity: critical, team: checkout, job: checkout }

promtool check rules rules/*.yml
promtool test rules tests/*.yml
python3 scripts/alert_lint.py rules/*.yml        # needs PyYAML, or pass JSON:
yq -o=json rules/checkout.yml | python3 scripts/alert_lint.py -

scripts/alert_lint.py checks every alerting rule for missing for, missing or unknown severity, missing summary/runbook_url, irate in alerts, raw counters without rate/increase, and recording-rule names that break the level:metric:ops convention. It exits 1 on any error-level finding.

6. Alertmanager

route:
  receiver: default-slack
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: pagerduty-oncall
      continue: true                    # also post to slack
    - matchers: [severity="critical"]
      receiver: team-slack
    - matchers: [severity="warning"]
      receiver: ticket-queue
      repeat_interval: 24h
inhibit_rules:
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity=~"warning|critical"]
    equal: [cluster]
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, service]
receivers:
  - name: pagerduty-oncall
    pagerduty_configs: [{ routing_key_file: /etc/alertmanager/pd_key }]
  - name: team-slack
    slack_configs: [{ api_url_file: /etc/alertmanager/slack_url, channel: "#checkout-alerts", send_resolved: true }]
  - name: default-slack
    slack_configs: [{ api_url_file: /etc/alertmanager/slack_url, channel: "#alerts" }]
  - name: ticket-queue
    webhook_configs: [{ url: http://ticket-bridge:8080/alert }]

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=critical service=checkout
amtool silence add alertname=CheckoutHighErrorRatio --duration=2h --comment="deploy rollback in progress"
amtool alert add TestAlert severity=warning service=checkout   # end-to-end test (non-prod or announced)

Prometheus Operator: rules ship as PrometheusRule objects whose labels must match the Prometheus ruleSelector; routing can be split with AlertmanagerConfig per namespace.

7. Review loop

After a week: list alerts by fire count and action rate, delete or retune the noisiest, and confirm the last real incident would have paged within 5 minutes.

Gotchas

rate() over a counter that resets is fine; delta() or subtraction on counters is wrong. increase() extrapolates and returns non-integers; do not alert on == 1.
sum(rate(...)) drops labels needed by routing (team, service); keep them in by (...).
An alert expression that returns no series is “OK”. Missing data never fires unless you add absent().
Prometheus 3 changed range selectors to left-open intervals and made UTF-8 metric and label names valid; re-run promtool test rules after upgrading from 2.x.
Templating {{ $value }} with high-precision floats produces ugly pages; use humanize, humanizePercentage, humanizeDuration.
Annotations should not contain changing values in labels; labels define alert identity, and a changing label creates a new alert each evaluation.
group_by: ['...'] disables grouping entirely and floods receivers.

Output

**Rules:** 6 recording, 9 alerting (4 critical pages, 5 warnings to tickets)
**Removed:** HighCPU, PodRestarted (cause alerts, 7% action rate)
**Tests:** promtool test rules 14/14 pass; alert_lint 0 errors
**Alertmanager:** critical -> PagerDuty + team Slack; warnings -> tickets; cluster-down inhibits
**E2E:** test alert routed to #checkout-alerts and PD test service at 14:02 UTC

Bundled files

Loaded only when a step in the skill needs them. Review them here before installing.

scripts/alert_lint.py 110 lines · 4.2 KB

#!/usr/bin/env python3
"""Lint Prometheus rule files for alert-quality problems promtool does not catch.

Usage:
  python3 scripts/alert_lint.py rules/*.yml          # YAML input needs PyYAML
  yq -o=json rules.yml | python3 scripts/alert_lint.py -   # JSON on stdin, stdlib only
  python3 scripts/alert_lint.py rules.json           # JSON file, stdlib only

Accepts plain rule files ({"groups": [...]}) and PrometheusRule CRDs ({"spec": {"groups": [...]}}).
Exit code 1 when any ERROR finding exists, 0 otherwise.
"""
import json
import re
import sys

SEVERITIES = {"critical", "warning", "info", "page", "ticket"}
RECORD_NAME = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]*:[a-zA-Z0-9_]+:[a-zA-Z0-9_]+$")
# metric name ending in _total used without rate/increase/irate/resets/changes wrapper
RAW_COUNTER = re.compile(r'(?<![\w:])([a-zA-Z_:][\w:]*_total)\b(?!\s*\[)')


def load(path):
    text = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read()
    try:
        return [json.loads(text)]
    except json.JSONDecodeError:
        pass
    try:
        import yaml  # type: ignore
    except ImportError:
        sys.exit(f"{path}: not JSON and PyYAML is not installed; pipe through `yq -o=json`")
    return [d for d in yaml.safe_load_all(text) if d]


def groups_of(doc):
    if isinstance(doc, dict):
        if "groups" in doc:
            return doc["groups"] or []
        if isinstance(doc.get("spec"), dict):
            return doc["spec"].get("groups") or []
    return []


def counter_without_rate(expr):
    hits = []
    expr = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)  # ignore label values in quotes
    for m in RAW_COUNTER.finditer(expr):
        start = m.start()
        prefix = expr[max(0, start - 40):start]
        # inside a range selector that is wrapped by a function is fine; only flag bare use
        if re.search(r"(rate|increase|irate|resets|changes|absent|absent_over_time)\(\s*$", prefix):
            continue
        hits.append(m.group(1))
    return hits


def lint(path):
    findings = []
    for doc in load(path):
        for g in groups_of(doc):
            gname = g.get("name", "?")
            for r in g.get("rules") or []:
                expr = str(r.get("expr", ""))
                if "record" in r:
                    name = r["record"]
                    if not RECORD_NAME.match(name):
                        findings.append(("WARN", gname, name, "recording rule name should be level:metric:operations"))
                    continue
                name = r.get("alert", "?")
                labels = r.get("labels") or {}
                ann = r.get("annotations") or {}
                add = lambda lvl, msg: findings.append((lvl, gname, name, msg))
                if not r.get("for"):
                    add("WARN", "no 'for' duration: a single bad scrape pages someone")
                sev = labels.get("severity")
                if not sev:
                    add("ERROR", "missing severity label (routing depends on it)")
                elif str(sev) not in SEVERITIES:
                    add("WARN", f"unusual severity '{sev}'; expected one of {sorted(SEVERITIES)}")
                if not ann.get("summary"):
                    add("ERROR", "missing summary annotation")
                if not ann.get("runbook_url"):
                    add("ERROR", "missing runbook_url annotation")
                if "irate(" in expr:
                    add("WARN", "irate() in an alert is spiky; use rate()")
                for c in counter_without_rate(expr):
                    add("ERROR", f"counter {c} used without rate()/increase()")
                for k, v in labels.items():
                    if "{{" in str(v) and "$value" in str(v):
                        add("ERROR", f"label {k} templates $value: alert identity changes every evaluation")
    return findings


def main(argv):
    if len(argv) < 2:
        print(__doc__)
        return 2
    total_err = 0
    count = 0
    for path in argv[1:]:
        for lvl, group, name, msg in lint(path):
            count += 1
            total_err += lvl == "ERROR"
            print(f"{lvl:5} {path} [{group}] {name}: {msg}")
    print(f"\n{count} finding(s), {total_err} error(s)")
    return 1 if total_err else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

About this skill

Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published Oct 20, 2026, last updated Oct 20, 2026. MIT licensed.

Install

no restart needed

curl -fsSL https://claudeskills.page/install.sh | sh -s -- prometheus-alerting

Installs to ~/.claude/skills/ — available in every project.

curl -fsSL https://claudeskills.page/install.sh | sh -s -- --project prometheus-alerting

Installs to ./.claude/skills/ — commit it to share with your team.

curl -fsSLO https://claudeskills.page/downloads/prometheus-alerting.zip
unzip -o prometheus-alerting.zip -d ~/.claude/skills/

Download and unzip yourself. Swap the path for .claude/skills/ to scope it to a repo.

& ([scriptblock]::Create((irm https://claudeskills.page/install.ps1))) prometheus-alerting

PowerShell 5.1+. Installs to %USERPROFILE%\.claude\skills\.

.zip

On Claude.ai, upload the zip under Settings → Skills. All methods

Archive contents

prometheus-alerting/
SKILL.md
scripts/alert_lint.py

What Claude sees before loading it

Try asking

Use it when

Skip it when

How to check the result

The common mistake it prevents

~/.claude/skills/prometheus-alerting/SKILL.md

Ground rules

Workflow

1. Inventory

2. Page, ticket, or dashboard

3. PromQL patterns and recording rules

4. Alert rules

5. Test and lint

6. Alertmanager

7. Review loop

Gotchas

Output

Bundled files

About this skill

Related skills

SLOs and Error Budgets

Runbooks

Grafana Dashboards

Incident Response

Firebase Backend

GitHub Actions