Content Moderation skill for Claude Code
Policy first, thresholds from data, humans for the gray zone. Claude starts moderation work from a written policy with near-miss examples, not from a toxicity score. It builds a labeled evaluation set, layers normalization, blocklists, a moderation API, an LLM judge and a human queue, sets thresholds per category from precision and recall targets, and adds appeals, statements of reason and an audit log that records the policy version behind each decision.
- name
content-moderation - size224 lines · ≈3,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Builds moderation for user-generated and model-generated content - a written policy taxonomy with severities and actions, a layered pipeline of normalization, blocklists and hash matching, a moderation API or classifier, an LLM judge for gray areas, and a human review queue, per-category thresholds chosen from a labeled evaluation set, appeals, statements of reason, and audit logging. Use when launching comments, chat, uploads or profiles, when adding input and output filtering to an LLM feature, when false positives or missed abuse pile up, or when preparing for trust-and-safety or DSA obligations.Try asking
- Add moderation to our comments: block the worst stuff, send borderline posts to a review queue
- Our filter keeps removing harmless posts. Tune the thresholds properly
- Filter both the prompts and the answers in our AI chat feature for policy violations
Use it when
- Launching comments, chat, uploads, profiles or any user-generated surface
- An LLM feature needs input and output policy filtering
- False positives, missed abuse or appeal volume are growing
- You need audit trails, notices and appeals for compliance
Skip it when
- The threat is untrusted text steering your agent or tools — use prompt-injection-defense
- You need a general-purpose LLM labeler for non-safety categories — use llm-classification
How to check the result
scripts/threshold_sweep.py on the frozen test split meets each category's precision and recall targets, shadow-mode decisions match a human-labeled sample, and appeal overturn rates stay low per category.
The common mistake it prevents
Running blocklists and classifiers on raw text without Unicode normalization or thread context, so obfuscated abuse passes while quotes and counter-speech get removed.
Moderation is done well when a written policy defines every category with examples, each category has its own thresholds chosen from a labeled evaluation set, automation acts only where precision is proven and sends the uncertain band to trained humans, every decision is logged with the policy version and reason, users can appeal, and precision, recall, review volume and appeal overturn rate are tracked per category. A single “toxicity > 0.8” switch is not moderation.
Ground rules
- Policy first, tooling second. If two reviewers cannot agree on a label from the written policy, no model will either.
- Separate the judgment (does this violate category X at severity Y?) from the action (remove, limit, label, warn, suspend). Actions change with surface and user history; definitions should not.
- No global threshold. Each category gets its own, from data.
- Run new rules and models in shadow mode (log, do not enforce) before turning them on.
- Child sexual abuse material has legal handling and reporting obligations (in the US, reports to NCMEC). Use industry hash matching, never store or forward samples for “testing”, and involve counsel before building this path.
- Protect reviewers: blur or grayscale media by default, cap exposure time, rotate queues.
Workflow
- [ ] 1. Write the policy taxonomy (categories, definitions, examples, severities, actions)
- [ ] 2. Map surfaces and timing (pre-publish block vs post-publish review)
- [ ] 3. Build the labeled evaluation set
- [ ] 4. Assemble the layered pipeline
- [ ] 5. Choose per-category thresholds and bands
- [ ] 6. Stand up the review queue, notices and appeals
- [ ] 7. Audit logging and metrics
- [ ] 8. Shadow, then enforce; re-evaluate on every model or policy change1. Policy taxonomy
Write it as a table, versioned in the repo (policy/v3.md):
| Category | Definition (one sentence) | Violating examples | Allowed examples (near misses) | Severity | Default action |
|---|---|---|---|---|---|
| harassment/targeted | Abuse directed at an identifiable person | "@sam you worthless idiot, quit" | Criticism of a public figure's policy; quoted abuse in a report | 2 | Remove + warn |
| violent-threat | Statement of intent to harm a person or group | "I know where you live, you're next" | Hyperbole in games/sports context | 3 | Remove + suspend + escalate |
| self-harm/intent | User expresses intent to self-harm | ... | Recovery stories, asking for help | 3 | Do NOT remove silently: show resources, route to specialist queue |
| spam/commercial | Unsolicited promotion, link farms | ... | Users sharing their own project once in context | 1 | Limit visibility |Rules: include near-miss allowed examples for every category (counter-speech, news reporting, reclaimed terms, quotes, fiction, education); define severity by harm, not by how offensive the words sound; list which categories require human review before action.
2. Surfaces and timing
| Surface | Timing | Why |
|---|---|---|
| Usernames, display names, avatars | Pre-publish, synchronous | Permanent, high visibility |
| Public posts, comments | Pre-publish for high-severity categories, async for the rest | Latency vs harm |
| Direct messages | Async plus user reporting; respect privacy commitments | Expectations differ |
| Uploads (images, video) | Hash match synchronously, classifier async, hold for review on high scores | Cost and latency |
| LLM feature input | Before the model call | Stop disallowed requests early |
| LLM feature output | Before display; for streams, check in chunks and stop on a hit | Model can produce what policy forbids |
3. Evaluation set
- 200+ items per major category with at least 30 confirmed violations each, sampled from your real traffic (stratified so rare categories are present), labeled by two reviewers against the written policy, disagreements adjudicated.
- Measure inter-rater agreement (Cohen’s kappa); below about 0.6 means the policy text is ambiguous. Fix the policy before tuning models.
- Include hard negatives: counter-speech, news, quotes, reclaimed slurs, dialects, identity terms used neutrally (“I am gay”), medical and educational content.
- Include adversarial variants: leetspeak, spacing, homoglyphs, zero-width characters, text inside images, emoji substitution, misspellings, other languages.
- Freeze a test split. Never tune thresholds on it.
4. Layered pipeline
| Layer | What | Decides |
|---|---|---|
| L0 normalize | Unicode NFKC, strip zero-width and control characters, fold homoglyphs, lowercase a copy for matching | Nothing; feeds later layers |
| L1 rules | Exact and regex blocklists, known-bad URL and domain lists, perceptual hash matching for known media, rate and account-age signals | Clear-cut cases only |
| L2 classifier | Moderation API or self-hosted classifier, scores per category | Auto-action above the precision threshold; below the review threshold, allow |
| L3 LLM judge | A capable model given the policy text and context (thread, reply-to) | Gray band between thresholds; outputs category, severity, policy clause, confidence |
| L4 human review | Trained reviewers with the policy and full context | Everything the judge is unsure about, every severe action, all appeals |
L2 options:
- OpenAI moderation endpoint, model
omni-moderation-latest: free, text and images, 13 categories (harassment,harassment/threatening,hate,hate/threatening,illicit,illicit/violent,self-harm,self-harm/intent,self-harm/instructions,sexual,sexual/minors,violence,violence/graphic); returnsflagged,categories,category_scores,category_applied_input_types. Only six categories score images; text-only categories return 0 for image-only input. - Open-weight safety classifiers you host (the Llama Guard family;
gpt-oss-safeguard, which takes your policy text at inference time) when data cannot leave your network or your categories do not match a vendor’s. - Do not build on Google’s Perspective API: it stops serving after December 31, 2026.
Map vendor categories onto your taxonomy explicitly. Vendor categories never match your policy exactly; unmapped policy categories need L1 rules or the L3 judge.
from openai import OpenAI
client = OpenAI()
resp = client.moderations.create(model="omni-moderation-latest", input=normalized_text)
r = resp.results[0]
scores = r.category_scores.model_dump(by_alias=True) # {"harassment": 0.91, "self-harm/intent": 0.02, ...}L3 judge prompt shape (use structured outputs so the verdict is parseable):
You apply the content policy below. Judge ONLY against this policy.
<policy version="v3">...full category table with examples...</policy>
<context>parent post and the two previous replies</context>
<content>the item under review</content>
Return JSON: {"category": one of [...] or "none", "severity": 0-3,
"policy_clause": "<quoted line from the policy>", "confidence": "high"|"medium"|"low"}
Quoting, reporting on, or condemning a violation is not itself a violation.Send confidence != "high" and any severity-3 verdict to human review. The content
under review is untrusted input to the judge; keep the judge tool-less (see
prompt-injection-defense).
5. Thresholds and bands
Run scores for the eval set through the classifier, then sweep per category:
python3 scripts/threshold_sweep.py eval.csv --auto-precision 0.95 --review-recall 0.90 \
--override violent-threat:0.98:0.97 --override spam/commercial:0.90:0.80Input columns: item_id,category,score,label. For each category it reports the lowest
threshold meeting the auto-action precision target, the highest threshold meeting the
combined recall target, precision and recall at each, and the share of traffic landing
in the auto and review bands. Exit 1 means a target is unreachable with that classifier;
then add an L3 judge for the band, improve the policy, or accept more review volume.
Trade-offs to set with the product owner, per category:
| Category type | Favor | Because |
|---|---|---|
| Severe harm (threats, sexual content involving minors, self-harm instructions) | Recall; send more to review | A miss is far worse than review cost |
| Low harm, high volume (spam, mild profanity) | Precision for auto-action | False removals erode trust and flood appeals |
| Speech-sensitive (hate, harassment) | Precision for auto-removal, recall for review | Context-dependent; humans decide edge cases |
Review volume must fit reviewer capacity: daily items x review% <= reviewers x items/day.
If not, raise the review threshold for low-severity categories first.
6. Review queue, notices, appeals
- Prioritize the queue by severity x reach (followers, views) x age. Severity-3 items get a short SLA; set it with the team that staffs the queue.
- Show reviewers the policy clause, the model’s reason, and context; record their label and clause, not just approve/reject. Their labels feed the eval set.
- Double-review a sample weekly and hold calibration sessions on disagreements.
- Tell users what was actioned, which rule, and how to appeal. Under the EU Digital Services Act, hosting services owe a statement of reasons for restrictions (Art. 17) and online platforms must offer a free internal complaint system for at least six months after the decision (Art. 20). Confirm obligations with counsel.
- Appeals go to a different reviewer than the original. Track overturn rate per category; a high rate means the threshold or policy wording is wrong.
- For spam and evasion, keep notices generic so adversaries cannot probe thresholds.
7. Audit log and metrics
Log every decision (automated or human) as an append-only record:
{"decision_id": "dec_01J...", "content_id": "cmt_88", "content_hash": "sha256:...",
"surface": "comment", "policy_version": "v3", "layer": "L2", "model": "omni-moderation-latest",
"scores": {"harassment": 0.97}, "category": "harassment/targeted", "severity": 2,
"action": "remove", "actor": "system", "reviewer_id": null, "appeal_of": null,
"created_at": "2026-10-20T09:12:44Z"}Store content by reference with its own retention period; the decision log outlives the content. Dashboards per category: volume, auto-action rate, review rate, review backlog age, precision from weekly human audits of auto-actions, appeal rate, overturn rate.
8. Shadow, enforce, re-evaluate
Run the pipeline in shadow for one to two weeks, compare its decisions with human labels on a sample, then enforce category by category. Re-run the eval and the threshold sweep whenever the vendor model, your prompt, or the policy version changes; scores shift between model versions.
Gotchas
- Blocklists without normalization miss
i d i o tand Cyrillic look-alikes; blocklists without word boundaries hit innocent words (the Scunthorpe problem). - Moderating a reply without its parent misreads quotes, sarcasm and counter-speech.
- Edited content and changed profile fields must be re-moderated.
- Text inside images and video frames bypasses text moderation; OCR it.
- Identity terms used neutrally get flagged at higher rates by many classifiers; compare false positive rates across dialect and identity slices in the eval set.
- Category scores are not comparable across categories or model versions.
- Sending user content to a third-party moderation API requires a data processing agreement and a mention in the privacy notice.
- Self-harm content needs a supportive path (resources, specialist review), not silent deletion.
Output contract
## Policy taxonomy (version, categories, severities, actions)
## Pipeline: layers per surface, timing, vendor/model per layer, category mapping
## Thresholds (scripts/threshold_sweep.py output) and expected review volume vs capacity
## Review queue: priority rule, SLAs, calibration cadence
## Notices and appeals flow
## Audit log schema and dashboards
## Rollout: shadow results, enforcement orderBundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/threshold_sweep.py
#!/usr/bin/env python3
"""Pick per-category moderation thresholds from a labeled evaluation set.
Input CSV with a header and one row per (item, category):
item_id,category,score,label
c1,harassment,0.91,1
c2,harassment,0.12,0
c3,self-harm,0.40,1
score = classifier or moderation-API score in [0, 1]; label = 1 if human reviewers judged
the item a violation of that category under your policy, else 0.
For each category the script finds:
auto threshold - lowest score whose precision >= --auto-precision (auto-action band)
review threshold - highest score whose recall >= --review-recall (send-to-human band)
and prints precision/recall at both, plus the share of items that would land in each band.
Usage:
python3 threshold_sweep.py eval.csv
python3 threshold_sweep.py eval.csv --auto-precision 0.97 --review-recall 0.95 \
--override self-harm:0.99:0.98
Exit 0 if every category meets its targets, 1 if any cannot, 2 on bad input.
"""
import argparse
import csv
import sys
from collections import defaultdict
def metrics(rows, t):
tp = sum(1 for s, y in rows if s >= t and y == 1)
fp = sum(1 for s, y in rows if s >= t and y == 0)
fn = sum(1 for s, y in rows if s < t and y == 1)
p = tp / (tp + fp) if tp + fp else 1.0
r = tp / (tp + fn) if tp + fn else 1.0
flagged = (tp + fp) / len(rows)
return p, r, flagged
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("csv")
ap.add_argument("--auto-precision", type=float, default=0.95, help="min precision for automatic action (default 0.95)")
ap.add_argument("--review-recall", type=float, default=0.90, help="min recall for auto+review bands combined (default 0.90)")
ap.add_argument("--override", action="append", default=[],
help="per-category targets as category:auto_precision:review_recall (repeatable)")
ap.add_argument("--min-positives", type=int, default=30, help="warn when a category has fewer labeled violations")
args = ap.parse_args()
targets = {}
for o in args.override:
try:
cat, p, r = o.rsplit(":", 2)
targets[cat] = (float(p), float(r))
except ValueError:
print(f"error: bad --override {o!r}", file=sys.stderr)
return 2
data = defaultdict(list)
try:
with open(newline="", file=args.csv) as f:
reader = csv.DictReader(f)
missing = {"category", "score", "label"} - set(reader.fieldnames or [])
if missing:
print(f"error: missing columns {sorted(missing)}", file=sys.stderr)
return 2
for n, row in enumerate(reader, 2):
try:
s = float(row["score"])
y = int(row["label"])
if y not in (0, 1) or not 0 <= s <= 1:
raise ValueError("score must be 0-1 and label 0/1")
except ValueError as e:
print(f"error: line {n}: {e}", file=sys.stderr)
return 2
data[row["category"].strip()].append((s, y))
except OSError as e:
print(f"error: {e}", file=sys.stderr)
return 2
if not data:
print("error: no rows", file=sys.stderr)
return 2
failed = []
print(f"{'category':<24} {'n':>6} {'pos':>5} | {'auto_t':>6} {'P':>5} {'R':>5} {'auto%':>6} | "
f"{'review_t':>8} {'P':>5} {'R':>5} {'review%':>7}")
for cat in sorted(data):
rows = data[cat]
pos = sum(y for _, y in rows)
p_target, r_target = targets.get(cat, (args.auto_precision, args.review_recall))
candidates = sorted({s for s, _ in rows} | {1.0})
auto_t = None
for t in candidates: # lowest t meeting precision = most recall
p, r, _ = metrics(rows, t)
if p >= p_target and r > 0:
auto_t = t
break
review_t = None
for t in reversed(candidates): # highest t meeting recall = least review load
p, r, _ = metrics(rows, t)
if r >= r_target:
review_t = t
break
line = f"{cat:<24} {len(rows):>6} {pos:>5} | "
if auto_t is None:
line += f"{'none':>6} {'-':>5} {'-':>5} {'-':>6} | "
failed.append(f"{cat}: no threshold reaches precision {p_target}")
else:
p, r, fl = metrics(rows, auto_t)
line += f"{auto_t:>6.3f} {p:>5.2f} {r:>5.2f} {fl*100:>5.1f}% | "
if review_t is None:
line += f"{'none':>8}"
failed.append(f"{cat}: no threshold reaches recall {r_target}")
else:
p, r, fl = metrics(rows, review_t)
band = fl - (metrics(rows, auto_t)[2] if auto_t is not None and auto_t > review_t else 0)
line += f"{review_t:>8.3f} {p:>5.2f} {r:>5.2f} {band*100:>6.1f}%"
if auto_t is not None and review_t > auto_t:
line += " (review_t > auto_t: auto band already meets recall)"
print(line)
if pos < args.min_positives:
print(f"{'':<24} warning: only {pos} labeled violations; thresholds are unstable - label more")
print("\nauto% = share of items auto-actioned; review% = share sent to human review "
"(between review_t and auto_t).")
if failed:
print("RESULT: targets not met -> " + "; ".join(failed))
return 1
print("RESULT: all categories meet their targets")
return 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.