Systematic Reviews skill for Claude Code

# Risk of bias and certainty: tool selection and domains

Read this at the bias-assessment and certainty steps. Use the official tool guidance
documents for signalling questions; this file summarizes structure so you pick the right
tool and record judgments consistently. Do not reproduce tool forms verbatim in outputs;
cite the tool.

## Tool selection

| Included design | Tool | Judgment levels |
|---|---|---|
| Randomized trials (parallel, cluster, crossover) | Cochrane RoB 2 (variants for cluster and crossover) | Low / Some concerns / High |
| Non-randomized studies of interventions | ROBINS-I (version 2 released 2025; check which version the protocol names) | Low / Moderate / Serious / Critical (V2 adds algorithm-suggested judgments) |
| Non-randomized studies of exposures | ROBINS-E | Low / Some concerns / High / Very high |
| Diagnostic accuracy | QUADAS-2 (QUADAS-C for comparative) | Low / High / Unclear, plus applicability |
| Prognostic factor studies | QUIPS | Low / Moderate / High |
| Prediction model studies | PROBAST | Low / High / Unclear |
| Qualitative studies | CASP qualitative checklist or JBI checklist | Methodological limitations (feeds GRADE-CERQual) |
| Systematic reviews (umbrella review) | AMSTAR 2 or ROBIS | Confidence: High / Moderate / Low / Critically low |
| Animal studies | SYRCLE RoB | Low / High / Unclear |
| Cross-sectional / prevalence | JBI prevalence checklist | Checklist items, summarized |

## RoB 2 domains (randomized trials)

1. Bias arising from the randomization process
2. Bias due to deviations from intended interventions (specify: effect of assignment
   [ITT] or effect of adhering)
3. Bias due to missing outcome data
4. Bias in measurement of the outcome
5. Bias in selection of the reported result

Overall: Low only if all domains are low; High if any domain is high or several have
some concerns that together substantially lower confidence. Assess per outcome, not per
study.

## ROBINS-I domains (non-randomized interventions)

Pre-intervention: confounding; selection of participants. At intervention:
classification of interventions. Post-intervention: deviations from intended
interventions; missing data; measurement of outcomes; selection of the reported result.
Requires a pre-specified list of important confounders and a "target trial" description
in the protocol. Version 2 also addresses immortal time bias explicitly.

## QUADAS-2 domains (diagnostic accuracy)

Patient selection; index test; reference standard; flow and timing. Each has risk of
bias; the first three also have applicability concerns.

## Recording format

One row per study per outcome. Every judgment needs a supporting quote or location.

```markdown
| Study | Outcome | D1 | D2 | D3 | D4 | D5 | Overall | Support (quotes / page refs) |
|-------|---------|----|----|----|----|----|---------|------------------------------|
| Lee 2021 | Pain at 12 wk | Low | Some | Low | High | Low | High | D4: unblinded self-report (p.5); D2: 18% crossover (Table 2) |
```

Two reviewers assess independently; record disagreements and how they were resolved.
Visualize with robvis (R package or web app) as traffic-light and summary plots.

## GRADE certainty of evidence

Start: randomized trials at High; observational studies at Low (with ROBINS-I, start at
High and let the tool's judgments rate down).

Rate down (one or two levels each):
- Risk of bias
- Inconsistency (unexplained heterogeneity; I^2 high, non-overlapping CIs)
- Indirectness (population, intervention, comparator or outcome differ from the question)
- Imprecision (CI crosses decision thresholds; optimal information size not met)
- Publication bias (small-study effects, funnel asymmetry with >= 10 studies, industry-only data)

Rate up (observational evidence without serious limitations):
- Large magnitude of effect
- Dose-response gradient
- All plausible residual confounding would reduce the observed effect

Final levels: High, Moderate, Low, Very low. Present in a Summary of Findings table:

```markdown
| Outcome | No. studies (participants) | Relative effect (95% CI) | Absolute effect | Certainty | Reasons |
|---------|----------------------------|---------------------------|-----------------|-----------|---------|
| Mortality 30 d | 6 RCTs (2,140) | RR 0.82 (0.69-0.97) | 21 fewer per 1,000 | Moderate | Imprecision (-1) |
```

Plain-language statements by certainty: High "reduces", Moderate "probably reduces",
Low "may reduce", Very low "the evidence is very uncertain about the effect".

## Meta-analysis checks

- Random-effects by default when studies differ clinically; report tau^2 and a
  prediction interval, not just I^2.
- Do not pool when designs, outcomes, or measures are incompatible; use structured
  narrative synthesis (SWiM reporting guideline) instead.
- Funnel plots and Egger's test only with 10 or more studies.
- Pre-specify subgroup and sensitivity analyses (e.g., excluding high risk-of-bias studies).

#!/usr/bin/env python3
"""Compute PRISMA 2020 flow-diagram counts from a screening log CSV and check it
for inconsistencies (missing decisions, decisions after exclusion, full-text
exclusions without a reason).

Usage: python3 prisma_flow.py screening.csv [--json]

Expected columns (header row required, case-insensitive):
  record_id         unique id per record
  source            database or register name (e.g. "PubMed", "register:CT.gov"),
                    or "other:<method>" (citation searching, websites, experts)
  duplicate         yes/no  - removed as duplicate before screening
  auto_excluded     yes/no  - optional; marked ineligible by automation tools
  title_abstract    include/exclude (blank for other-method records)
  retrieved         yes/no  - full text obtained (for records passing screening)
  full_text         include/exclude
  exclusion_reason  required when full_text = exclude
  study_id          optional; groups multiple reports of one study

Exit status: 0 consistent, 1 inconsistencies found, 2 usage error.
"""
from __future__ import annotations

import csv
import json
import sys
from collections import Counter

YES = {"yes", "y", "true", "1"}
NO = {"no", "n", "false", "0", ""}


def norm(v: str | None) -> str:
    return (v or "").strip().lower()


def flow(rows: list[dict]) -> tuple[dict, list[str]]:
    errs: list[str] = []
    ids = Counter(norm(r.get("record_id")) for r in rows)
    for rid, c in ids.items():
        if c > 1:
            errs.append(f"record_id '{rid}' appears {c} times")

    def pathway(r: dict) -> str:
        return "other" if norm(r.get("source")).startswith("other") else "db"

    out = {p: Counter() for p in ("db", "other")}
    by_source: Counter = Counter()
    reasons = {"db": Counter(), "other": Counter()}
    included_studies: dict[str, set] = {"db": set(), "other": set()}

    for r in rows:
        rid = r.get("record_id", "?")
        p = pathway(r)
        c = out[p]
        c["identified"] += 1
        by_source[r.get("source", "").strip() or "(blank)"] += 1
        dup, auto = norm(r.get("duplicate")), norm(r.get("auto_excluded"))
        ta, ret, ft = norm(r.get("title_abstract")), norm(r.get("retrieved")), norm(r.get("full_text"))
        for col, val, allowed in (("duplicate", dup, YES | NO), ("auto_excluded", auto, YES | NO),
                                  ("retrieved", ret, YES | NO),
                                  ("title_abstract", ta, {"include", "exclude", ""}),
                                  ("full_text", ft, {"include", "exclude", ""})):
            if val not in allowed:
                errs.append(f"{rid}: invalid {col} value '{val}'")
        if dup in YES:
            c["removed_duplicates"] += 1
            if ta or ft:
                errs.append(f"{rid}: duplicate but has screening decisions")
            continue
        if auto in YES:
            c["removed_automation"] += 1
            if ta or ft:
                errs.append(f"{rid}: automation-excluded but has screening decisions")
            continue
        if p == "db":
            c["screened"] += 1
            if not ta:
                errs.append(f"{rid}: missing title_abstract decision")
                continue
            if ta == "exclude":
                c["excluded_screening"] += 1
                if ret in YES or ft:
                    errs.append(f"{rid}: excluded at screening but has full-text data")
                continue
        c["sought"] += 1
        if ret not in YES:
            c["not_retrieved"] += 1
            if ft:
                errs.append(f"{rid}: full_text decision without retrieved=yes")
            continue
        c["assessed"] += 1
        if not ft:
            errs.append(f"{rid}: retrieved but missing full_text decision")
            continue
        if ft == "exclude":
            reason = (r.get("exclusion_reason") or "").strip()
            if not reason:
                errs.append(f"{rid}: full-text exclusion without exclusion_reason")
                reason = "(no reason recorded)"
            reasons[p][reason] += 1
            continue
        c["included_reports"] += 1
        included_studies[p].add((r.get("study_id") or "").strip() or f"rec:{rid}")

    for p in out:
        out[p]["included_studies_pathway"] = len(included_studies[p])
    total_studies = len(included_studies["db"] | included_studies["other"])
    total_reports = out["db"]["included_reports"] + out["other"]["included_reports"]
    result = {
        "identified_by_source": dict(by_source),
        "databases_registers": dict(out["db"]),
        "other_methods": dict(out["other"]),
        "full_text_exclusion_reasons": {k: dict(v) for k, v in reasons.items()},
        "total_included_studies": total_studies,
        "total_included_reports": total_reports,
    }
    return result, errs


def render(res: dict) -> str:
    d, o = res["databases_registers"], res["other_methods"]
    g = lambda c, k: c.get(k, 0)
    lines = ["PRISMA 2020 flow counts", "", "IDENTIFICATION"]
    for s, n in sorted(res["identified_by_source"].items()):
        lines.append(f"  {s}: n = {n}")
    lines += [
        f"  Records identified (databases/registers): n = {g(d, 'identified')}",
        f"  Records removed before screening: duplicates n = {g(d, 'removed_duplicates')}, "
        f"automation n = {g(d, 'removed_automation')}",
        f"  Records identified (other methods): n = {g(o, 'identified')}"
        f" (duplicates removed n = {g(o, 'removed_duplicates')})",
        "", "SCREENING (databases/registers | other methods)",
        f"  Records screened: n = {g(d, 'screened')}",
        f"  Records excluded: n = {g(d, 'excluded_screening')}",
        f"  Reports sought for retrieval: n = {g(d, 'sought')} | n = {g(o, 'sought')}",
        f"  Reports not retrieved: n = {g(d, 'not_retrieved')} | n = {g(o, 'not_retrieved')}",
        f"  Reports assessed for eligibility: n = {g(d, 'assessed')} | n = {g(o, 'assessed')}",
    ]
    for p, label in (("db", "databases/registers"), ("other", "other methods")):
        rs = res["full_text_exclusion_reasons"][p]
        if rs:
            lines.append(f"  Reports excluded ({label}): n = {sum(rs.values())}")
            for reason, n in sorted(rs.items(), key=lambda x: -x[1]):
                lines.append(f"    {reason}: n = {n}")
    lines += ["", "INCLUDED",
              f"  Studies included in review: n = {res['total_included_studies']}",
              f"  Reports of included studies: n = {res['total_included_reports']}"]
    return "\n".join(lines)


def main(argv: list[str]) -> int:
    args = [a for a in argv if not a.startswith("--")]
    if len(args) != 1:
        print(__doc__.strip(), file=sys.stderr)
        return 2
    try:
        with open(args[0], newline="", encoding="utf-8-sig") as fh:
            reader = csv.DictReader(fh)
            reader.fieldnames = [f.strip().lower() for f in (reader.fieldnames or [])]
            rows = list(reader)
    except OSError as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 2
    missing = {"record_id", "source", "duplicate", "title_abstract", "retrieved", "full_text"} - set(reader.fieldnames)
    if missing:
        print(f"error: missing columns: {', '.join(sorted(missing))}", file=sys.stderr)
        return 2
    res, errs = flow(rows)
    if "--json" in argv:
        print(json.dumps({**res, "errors": errs}, indent=2))
    else:
        print(render(res))
        if errs:
            print(f"\n{len(errs)} inconsistency(ies):")
            for e in errs:
                print(f"  - {e}")
        else:
            print("\nLog is internally consistent.")
    return 1 if errs else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))