Qualitative Coding skill for Claude Code
Codebooks, kappa checks and themes backed by quotes. Claude picks reflexive or codebook thematic analysis to match the study, builds codes with definitions, include and exclude rules and counter-examples, and checks double-coded samples with a bundled kappa script that names the exact disagreeing segments. Themes are written as claims, tested against negative cases, and every excerpt is verbatim with a source ID.
- name
qualitative-coding - size163 lines · ≈2,000 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Codes qualitative data rigorously — interview transcripts, open-ended survey answers, support tickets, field notes — using reflexive thematic analysis or a structured codebook, with inductive and deductive codes, code definitions with inclusion and exclusion rules, double-coding and Cohen's kappa checks, theme development, and an audit trail linking every theme to quotes. Use when asked to code transcripts, build a codebook, run thematic analysis, analyze open-text responses, measure inter-rater reliability, or turn qualitative data into themes.Try asking
- Code these 14 interview transcripts and build a codebook we can reuse
- Two of us coded the same 50 survey comments, check our inter-rater reliability
- Turn these 800 open-text NPS comments into themes with supporting quotes
Use it when
- You have transcripts, field notes or open-text answers to analyze
- You need a codebook a team can apply consistently
- You must report inter-rater reliability
- You want LLM-assisted coding with a human verification step
Skip it when
- You want product insights and recommendations from research rather than the coding method — use user-research-synthesis
- You are still writing the questions — use survey-design
How to check the result
coder_agreement.py shows every code at or above the agreed kappa on a fresh double-coded sample, and each theme cites verbatim excerpts from several sources.
The common mistake it prevents
Naming themes after interview topics, which summarizes what was asked instead of what participants meant.
--- name: qualitative-coding description: Codes qualitative data rigorously — interview transcripts, open-ended survey answers, support tickets, field notes — using reflexive… license: MIT ---
Done well means: every code has a definition someone else could apply, every theme is a patterned claim supported by data from multiple sources (not a topic label), each claim links to verbatim excerpts with IDs, negative cases are reported, and the method chosen matches the epistemology (reflexive vs. reliability-oriented).
Hard rules
- Quote verbatim only. Never paraphrase inside quotation marks, merge two speakers, or
invent a quote to illustrate a theme. Every excerpt carries a source ID
(
P07, 00:14:32orticket #4812). - Report prevalence honestly: “6 of 14 participants”, never “most users” from 3 people. For reflexive thematic analysis, avoid counting as evidence of importance; say why a theme matters.
- Anonymize before analysis: replace names, employers, locations, and rare details that re-identify people. Keep the key file separate from the data.
- Keep an audit trail: codebook versions with dates and the reason for each change.
Choose the approach
| Situation | Approach | Reliability check |
|---|---|---|
| Exploratory, interpretive, single analyst, academic | Reflexive thematic analysis (Braun & Clarke) | None statistically; reflexive memo and transparency instead |
| Team coding, product research, large volume, need consistency | Codebook thematic analysis / framework analysis | Double-code 10-20% of segments; Cohen’s kappa per code |
| Testing an existing theory or framework | Deductive (a priori) codebook, open to new codes | Kappa on the a priori codes |
| Building theory from the data | Grounded theory (open, axial, selective coding; constant comparison) | Memos, theoretical sampling, saturation argument |
| Thousands of short open-text answers | Codebook + LLM-assisted first pass + human verification sample | Kappa between human and model on a held-out sample |
Do not compute kappa for reflexive thematic analysis; Braun and Clarke explicitly reject it for that approach. Do report it for codebook and team approaches.
Workflow
- [ ] 1. Prepare: transcribe, anonymize, segment, assign IDs
- [ ] 2. Familiarize: read everything once; memo first impressions
- [ ] 3. Initial coding: inductive (and deductive if a framework applies)
- [ ] 4. Build the codebook from the initial codes
- [ ] 5. Double-code a sample; compute agreement; refine; recode
- [ ] 6. Code the full dataset
- [ ] 7. Develop themes: cluster codes, test against data, name them
- [ ] 8. Check negative cases and saturation
- [ ] 9. Write up with excerpt evidence1. Prepare
- Segment unit: a complete thought (one answer, one paragraph of a response, one ticket). Keep segments consistent across coders; agreement is meaningless if coders segment differently.
- Store as a table:
segment_id, source_id, speaker, text, metadata (role, plan, date). Metadata enables later comparisons (by segment, cohort, time).
3. Initial coding
- Code close to the data first (descriptive, in-vivo codes using participants’ words), then move to interpretive codes.
- Code for what is said and what it means: “pricing page” is a topic; “can’t predict monthly cost” is a code that carries meaning.
- Apply as many codes as fit; multi-labeling is normal.
- Write a memo whenever something surprises you or contradicts an earlier code.
4. Codebook
| Code | Definition | Include when | Exclude when (use instead) | Example (ID) | Counter-example (ID) |
|------|------------|--------------|-----------------------------|--------------|----------------------|
| cost_unpredictability | Participant cannot estimate or anticipate what they will pay | Mentions surprise bills, usage-based fear, budgeting difficulty | General "too expensive" without uncertainty (use price_level) | "I never know if this month is $40 or $400" (P03) | "It's pricey for what it does" (P11 -> price_level) |- 15-40 codes is typical for a project; more than ~60 usually means codes should be merged or organized into parent codes.
- Version the codebook (
codebook_v3_2025-10-14.md) and log each change: merged, split, redefined, with reason.
5. Inter-coder agreement (codebook approaches)
- Two coders independently code the same 10-20% of segments (stratified across sources) using the same segmentation.
- Export as long CSV:
segment_id,coder,code(one row per applied code; a row with an empty code for segments with none). - Run:
python3 scripts/coder_agreement.py double_coded.csv # exits 1 if any code < 0.6
python3 scripts/coder_agreement.py double_coded.csv --threshold 0.7 --jsonThe script computes Cohen’s kappa per code on presence/absence across the shared segments, percent agreement, Landis-Koch strength bands, and lists the segments where coders disagree.
- Interpret: kappa above 0.8 almost perfect; 0.6-0.8 acceptable for most applied work; < 0.6 means the definition is ambiguous. Discuss the listed disagreements, sharpen the include/exclude rules, and recode a fresh sample. Do not just negotiate the old one into agreement.
- Rare codes (applied to < 5% of segments) produce unstable kappa; report percent agreement and raw counts alongside, and consider merging.
For more than two coders or missing data, use Krippendorff’s alpha (R irr or
krippendorff Python package) instead of pairwise kappa.
7. Develop themes
- A theme is a central organizing idea with a claim: “Users trust the product less when costs are unpredictable, and route around it with spreadsheets” — not “Pricing”.
- Cluster related codes; draw a thematic map (theme -> subthemes -> codes).
- Review each candidate theme twice: against its coded extracts (coherent?) and against the whole dataset (does it hold across sources, or is it one talkative participant?).
- Note which segments or cohorts express a theme and which do not.
8. Negative cases and saturation
- Actively search for data that contradicts each theme; report it and explain whether it bounds the theme (e.g., “except admins on annual contracts”).
- Saturation (codebook approaches): track new codes per additional transcript; claim saturation only when 2-3 consecutive transcripts add no new codes, and say it is code saturation, not meaning saturation.
LLM-assisted coding
- Give the model the codebook (definitions, include/exclude, examples) and require
output as
segment_id, codes[], short rationalewith codes only from the codebook. - Human-code a random sample of at least 100 segments (or 10%) blind, then compute kappa human vs. model with the script (use “human” and “model” as coder names). Below 0.6: fix definitions or prompts; do not ship.
- Never let the model generate quotes; extract excerpts by segment ID from the source table.
Output template
# Findings: <study> (n = 14 interviews, 312 survey comments; coded 2025-10)
**Method:** codebook thematic analysis; 2 coders; 15% double-coded; mean kappa 0.74 (range 0.62-0.88)
## Theme 1: <claim sentence>
Prevalence: 9/14 participants; 61 comments; strongest among <segment>.
Evidence:
> "I never know if this month is $40 or $400." — P03 (ops lead, SMB)
> "We export usage into a sheet every Friday just to be safe." — P09
Negative cases: annual-contract admins (P05, P12) did not report this.
Implication: ...
## Theme 2 ...
## Codebook (appendix, final version) and change log
## Limitations: sample, coder positionality, segments not double-codedGotchas
- Themes that mirror the interview guide’s questions (“Theme 1: Onboarding”) are topic summaries, not analysis.
- High percent agreement with low kappa means the code is rare or near-universal; chance agreement is doing the work.
- Coding while still interviewing biases later interviews toward early codes; memo it if unavoidable.
- One articulate participant can dominate excerpts; spread evidence across sources.
Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/coder_agreement.py
#!/usr/bin/env python3
"""Inter-coder agreement for multi-label qualitative coding (two coders).
Input: a long-format CSV with header segment_id,coder,code
- one row per (segment, coder, code applied)
- a segment a coder reviewed but gave no code: row with an empty code
Every segment reviewed by both coders is compared; for each code the presence/
absence decision is treated as a binary rating and Cohen's kappa is computed.
Usage: python3 coder_agreement.py codes.csv [--threshold 0.6] [--json]
Exit status: 0 all codes >= threshold, 1 some codes below threshold, 2 bad input.
"""
from __future__ import annotations
import argparse
import csv
import json
import sys
from collections import defaultdict
def kappa(a: list[int], b: list[int]) -> float | None:
n = len(a)
if n == 0:
return None
po = sum(x == y for x, y in zip(a, b)) / n
pa, pb = sum(a) / n, sum(b) / n
pe = pa * pb + (1 - pa) * (1 - pb)
if pe == 1:
return None # both coders constant and identical: kappa undefined
return (po - pe) / (1 - pe)
def label(k: float | None) -> str:
if k is None:
return "undefined"
# Landis & Koch (1977) bands
for cut, name in ((0.8, "almost perfect"), (0.6, "substantial"), (0.4, "moderate"), (0.2, "fair"), (0.0, "slight")):
if k > cut:
return name
return "poor"
def main(argv: list[str]) -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("csv")
ap.add_argument("--threshold", type=float, default=0.6)
ap.add_argument("--json", action="store_true")
a = ap.parse_args(argv)
try:
with open(a.csv, newline="", encoding="utf-8-sig") as fh:
rows = list(csv.DictReader(fh))
except OSError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
if not rows or not {"segment_id", "coder", "code"} <= set(rows[0]):
print("error: CSV needs columns segment_id,coder,code", file=sys.stderr)
return 2
applied: dict[str, dict[str, set]] = defaultdict(lambda: defaultdict(set))
codes: set[str] = set()
for r in rows:
seg, coder, code = r["segment_id"].strip(), r["coder"].strip(), (r["code"] or "").strip()
applied[coder][seg] # mark segment as reviewed
if code:
applied[coder][seg].add(code)
codes.add(code)
coders = sorted(applied)
if len(coders) != 2:
print(f"error: expected exactly 2 coders, found {len(coders)}: {coders}", file=sys.stderr)
return 2
c1, c2 = coders
shared = sorted(set(applied[c1]) & set(applied[c2]))
only = sorted(set(applied[c1]) ^ set(applied[c2]))
if not shared:
print("error: no segments coded by both coders", file=sys.stderr)
return 2
results = []
for code in sorted(codes):
x = [int(code in applied[c1][s]) for s in shared]
y = [int(code in applied[c2][s]) for s in shared]
k = kappa(x, y)
agree = sum(i == j for i, j in zip(x, y)) / len(shared)
disagreements = [s for s, i, j in zip(shared, x, y) if i != j]
results.append({"code": code, "kappa": None if k is None else round(k, 3), "strength": label(k),
"percent_agreement": round(agree, 3), "applied": {c1: sum(x), c2: sum(y)},
"disagreements": disagreements})
exact = sum(applied[c1][s] == applied[c2][s] for s in shared) / len(shared)
below = [r for r in results if r["kappa"] is not None and r["kappa"] < a.threshold]
if a.json:
print(json.dumps({"coders": coders, "segments_compared": len(shared), "segments_single_coder": only,
"exact_match_rate": round(exact, 3), "codes": results}, indent=2))
else:
print(f"Coders: {c1} vs {c2}; segments compared: {len(shared)}"
+ (f"; coded by one coder only (ignored): {len(only)}" if only else ""))
print(f"Segments with identical code sets: {exact:.0%}\n")
print(f"{'code':<28}{'kappa':>7} {'strength':<15}{'agree':>6} applied({c1}/{c2})")
for r in sorted(results, key=lambda r: (r["kappa"] is None, r["kappa"] if r["kappa"] is not None else 0)):
k = " n/a" if r["kappa"] is None else f"{r['kappa']:>7.2f}"
print(f"{r['code']:<28}{k} {r['strength']:<15}{r['percent_agreement']:>6.0%} "
f"{r['applied'][c1]}/{r['applied'][c2]}")
if below:
print(f"\n{len(below)} code(s) below kappa {a.threshold}: revise definitions, then recode:")
for r in below:
print(f" {r['code']}: disagreements on segments {', '.join(r['disagreements'][:10])}"
+ (" ..." if len(r["disagreements"]) > 10 else ""))
return 1 if below else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.