Paper Analysis skill for Claude Code

Done well means: every claim in the paper is listed with the exact evidence offered for it, each weakness is specific enough that the authors could fix it, the verdict says how much the conclusions should move a reader’s beliefs, and nothing in the summary is absent from the paper.

Hard rules

• Work from the paper itself. If you only have the abstract or a title, say so and limit the analysis to what the abstract supports; do not reconstruct methods or numbers from memory.
• Quote or cite section/figure/table numbers for every factual statement about the paper (“Table 3 reports…”, “Section 4.2 states…”).
• Separate what the authors claim, what the evidence shows, and your judgment. Label each.
• Confirm the paper’s identity before analyzing: title, authors, year, venue, version (arXiv v1 vs. v3 can differ materially). Check retraction or erratum notices on the publisher page or via Crossref.
• Critique the work, never the authors. No speculation about motives.

Workflow

- [ ] 1. Identify the paper and version; note venue and peer-review status
- [ ] 2. First pass: abstract, figures, conclusion — write the claimed contribution
- [ ] 3. Claim map: each claim -> evidence -> strength
- [ ] 4. Method check: design, data, baselines, statistics
- [ ] 5. Threats to validity
- [ ] 6. Reproducibility check
- [ ] 7. Verdict and output in the requested format

2. Three-pass reading

1. Skim (5 min): title, abstract, intro’s last paragraph (contribution list), section headings, figures and tables, conclusion. Write one sentence: “The paper claims that X, shown by Y, in setting Z.”
2. Content pass: methods and results in full; note every number that supports a headline claim and where it comes from.
3. Adversarial pass: for each headline claim, ask what result would have falsified it and whether the authors ran that test.

3. Claim map

| # | Claim (paraphrased, with location) | Evidence offered | Type | Strength | Notes |
|---|------------------------------------|------------------|------|----------|-------|
| C1 | Method beats SOTA on 4 benchmarks (Abstract, Table 2) | Single-run accuracy, 4 datasets | Empirical | Moderate | No variance; baseline numbers copied from older papers |
| C2 | Gains come from the new attention module (Sec 5.1) | Ablation removing module | Causal | Weak | Ablation on 1 dataset only |

Strength scale: Strong (multiple independent tests, appropriate controls, uncertainty reported), Moderate (one adequate test or minor gaps), Weak (indirect, underpowered, confounded), Unsupported (asserted, cited elsewhere, or contradicted by the paper’s own data).

Flag claim inflation: abstract or title stating more than the results (causal language from correlational data, “generalizes” from one dataset, “significantly” without a test).

4. Method check by design

Design	Check first
Randomized trial / A/B test	Randomization and allocation concealment, pre-registration vs. reported outcomes, attrition, ITT vs. per-protocol, power
Observational / cohort	Confounders adjusted and how chosen, selection bias, reverse causation, immortal time bias, sensitivity analyses
ML / benchmark paper	Train/test leakage and contamination, tuned baselines with equal budget, seeds and variance, test-set reuse, compute reported
Survey study	Sampling frame, response rate, question wording, nonresponse bias, weighting
Qualitative	Sampling logic, saturation argument, reflexivity, audit trail, negative cases
Theory / proof	Assumptions stated, are they realistic, gaps between theorem and claimed implication
Systematic review / meta-analysis	Protocol registration, search completeness, risk-of-bias tool, heterogeneity (I^2), small-study effects

Statistics checklist:

• Effect sizes with confidence intervals, not only p-values. Is the effect practically meaningful?
• Multiple comparisons: many outcomes or subgroups tested without correction or pre-specification.
• p-values clustered just under 0.05, or outcomes switched relative to a registration.
• Sample size vs. claimed precision; tiny-n studies reporting large effects (winner’s curse).
• Unit of analysis errors (treating repeated measures or clustered data as independent).
• Percent improvements that hide tiny absolute differences (relative vs. absolute risk).

5. Threats to validity

Work through all four; write “none identified” only after checking.

• Internal: could something other than the treatment explain the result? (confounding, leakage, selection, measurement artifacts, experimenter degrees of freedom)
• Construct: do the measures capture the concept claimed? (benchmark vs. real capability, proxy outcomes, self-report)
• External: does it transfer beyond the sample, dataset, language, population, or time period studied?
• Statistical conclusion: power, variance, test choice, assumptions violated.

6. Reproducibility check

• Code, data, and model weights available? Licensed? Commit or version pinned?
• Enough detail to reimplement: hyperparameters, preprocessing, exclusion rules, seeds.
• Pre-registration or protocol link; does the paper deviate from it, and does it say so?
• Conflicts of interest and funding statement.

7. Verdict

Give one of: Conclusions supported, Supported with caveats, Partially supported (some claims hold, headline overreaches), Not supported. Add a confidence level (high/medium/low) and the single result that would most change your assessment.

Output templates

Pick the format the user asked for; default to the summary.

Structured summary

**Paper:** Title — Authors (Year), Venue, version/DOI. Peer-reviewed: yes/no.
**One-sentence claim:** ...
**Approach:** 2-3 sentences.
**Key results:** bullets with numbers and table/figure references.
**Strengths:** 2-4 bullets.
**Weaknesses:** 2-4 bullets, most important first.
**Verdict:** Supported with caveats (confidence: medium). Would change if ...
**Who should care:** ...

Referee report

Summary of the submission (neutral, 1 paragraph)
Major concerns (numbered; each: issue, why it matters, what would resolve it)
Minor concerns (numbered; clarity, typos, missing references)
Questions for the authors
Recommendation: accept / minor revision / major revision / reject, with one-line rationale

Major concerns are ones that could change the conclusions; everything else is minor. Each concern should request a concrete analysis, control, or clarification.

Journal-club brief: the summary plus 3 discussion questions and one figure to present, with what to look at in it.

Gotchas

• Do not penalize a paper for not doing a different study. Judge it against its stated question, then note scope limits separately.
• “Statistically significant” on a huge sample can be trivially small; on a tiny sample, “not significant” is not evidence of no effect.
• Benchmark SOTA claims often compare against baselines run with defaults while the proposed method was tuned. Check who ran the baselines and with what budget.
• Supplementary material often holds the real exclusions, failed runs, or extra seeds. Read it before calling something missing.
• When a claim depends on a cited prior result, check that the cited paper actually says it; citation distortion is common.