Statistical Testing skill for Claude Code

A test result is complete when the hypothesis and test were chosen before looking at the outcome, the test matches the data’s type, pairing, and independence, the report leads with the effect size and its confidence interval, multiple comparisons are corrected, and the conclusion is stated in plain language without “proves”. A p-value alone answers almost nothing a stakeholder asked.

Workflow

- [ ] 1. State the question as H0/H1, the primary outcome, alpha, and one- vs two-sided
- [ ] 2. Describe the data: type, groups, pairing, sample sizes, independence
- [ ] 3. Pick the test from the decision table
- [ ] 4. Check the assumptions that actually matter
- [ ] 5. Run the test; compute effect size and CI
- [ ] 6. Correct for multiple comparisons if more than one test
- [ ] 7. Report with the template; state limitations

1. Before computing anything

• Two-sided by default. One-sided only if the opposite direction is genuinely irrelevant and was declared beforehand.
• Alpha 0.05 unless the domain sets another; say so.
• Count the tests you will run (outcomes x subgroups x comparisons) - that is the family for correction.
• If the data are a full population (all orders last month), a significance test answers a hypothetical question; descriptive comparison may be enough. Say so.

3. Decision table

Outcome	Groups	Design	Default test	Alternative
Continuous	2	independent	Welch t-test (ttest_ind(equal_var=False))	Mann-Whitney U for ordinal or heavy outliers; bootstrap CI of the mean difference
Continuous	2	paired (before/after, same units)	paired t-test (ttest_rel)	Wilcoxon signed-rank
Continuous	3+	independent	Welch ANOVA, or one-way ANOVA if variances similar	Kruskal-Wallis; then post-hoc (Games-Howell / Tukey HSD / Dunn)
Continuous	3+	repeated measures	repeated-measures ANOVA or mixed model	Friedman
Binary / proportion	2	independent	two-proportion z-test or chi-square 2x2	Fisher exact when any expected count < 5
Binary	2	paired	McNemar	exact McNemar for small discordant counts
Categorical	r x c	independent	chi-square test of independence (chi2_contingency)	Fisher-Freeman-Halton / Monte Carlo for sparse tables
Counts per unit time	2	independent	Poisson / negative binomial rate comparison (GLM)	-
Two continuous variables	-	association	Pearson r (linear)	Spearman rho (monotonic, ordinal, outliers)
Outcome with covariates	any	adjust for confounders	regression (OLS / logistic / GLM) with HC3 standard errors	-
Time to event	2+	censored	log-rank test, Cox model	-

Prefer Welch over Student’s t always; it loses almost nothing when variances are equal. Mann-Whitney does not test medians in general - it tests whether one group tends to have larger values (stochastic dominance). Report it that way.

4. Assumptions that matter

Assumption	Matters for	Check	If violated
Independence of observations	every test	design: repeated users, clustered by store/classroom/session	aggregate to the unit, paired test, mixed model, or cluster-adjusted standard errors
Approximate normality of the mean	t-tests with small n	n per group; histograms/QQ plots	n above ~30-50 per group usually fine via CLT unless very skewed; else bootstrap or rank test
Expected cell counts >= 5	chi-square	chi2_contingency(...).expected_freq	Fisher exact or combine categories
Equal variances	Student t, classic ANOVA	ratio of SDs	use Welch versions
Linearity, no extreme influential points	Pearson, OLS	scatter plot	Spearman, transform, Huber regression

Do not run normality tests (Shapiro-Wilk) as a gate: with large n they reject trivial deviations, with small n they miss real ones. Look at plots and think about the data.

Independence is the assumption most often broken: 10,000 page views from 300 users are not 10,000 independent observations.

5. Run it

import numpy as np
from scipy import stats

a = np.asarray(group_a, float); b = np.asarray(group_b, float)

res = stats.ttest_ind(a, b, equal_var=False)
ci = res.confidence_interval(confidence_level=0.95)        # CI of mean difference (scipy >= 1.10)
diff = a.mean() - b.mean()

# Effect size: Hedges' g (bias-corrected Cohen's d)
na, nb = len(a), len(b)
sp = np.sqrt(((na - 1) * a.var(ddof=1) + (nb - 1) * b.var(ddof=1)) / (na + nb - 2))
g = diff / sp * (1 - 3 / (4 * (na + nb) - 9))

# Rank-based alternative with effect size (rank-biserial correlation)
u = stats.mannwhitneyu(a, b, alternative="two-sided")
rbc = 2 * u.statistic / (na * nb) - 1    # scipy reports U for `a`; positive means a tends to be larger

# Proportions
table = np.array([[conv_a, n_a - conv_a], [conv_b, n_b - conv_b]])
chi2, p, dof, expected = stats.chi2_contingency(table, correction=False)
odds, p_fisher = stats.fisher_exact(table)

# Bootstrap CI for a difference in medians (no distribution assumption)
boot = stats.bootstrap((a, b), lambda x, y: np.median(x) - np.median(y),
                       n_resamples=10_000, vectorized=False, random_state=0)

Test	Effect size to report
t-tests	mean difference with CI (in original units) + Hedges’ g
Mann-Whitney / Wilcoxon	rank-biserial correlation, or Hodges-Lehmann shift estimate
Proportions	absolute difference in percentage points with CI, plus relative lift or odds ratio
Chi-square r x c	Cramer’s V
ANOVA	eta-squared / omega-squared, plus pairwise differences with CIs
Correlation	r or rho with CI

Rough Cohen benchmarks (d 0.2 small, 0.5 medium, 0.8 large) are a last resort; prefer domain meaning (“3.1 minutes faster per ticket”).

Power and sample size before collecting data:

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.3, alpha=0.05, power=0.8, alternative="two-sided")

A non-significant result from an underpowered study is “inconclusive”, not “no effect”. For equivalence claims (“no meaningful difference”), run TOST with a pre-set margin.

6. Multiple comparisons

python3 scripts/adjust_pvalues.py 0.004 0.012 0.03 0.049 0.21          # Bonferroni, Holm, BH, BY side by side
python3 scripts/adjust_pvalues.py --file pvals.csv --method holm --alpha 0.05

Situation	Method
Few confirmatory tests, any false positive is costly	Holm (controls FWER; dominates Bonferroni)
Many exploratory tests (segments, metrics, genes), tolerate some false discoveries	Benjamini-Hochberg (FDR)
Many tests with arbitrary dependence	Benjamini-Yekutieli
Pairwise comparisons after ANOVA	Tukey HSD / Games-Howell, which handle the family internally

statsmodels equivalent: multipletests(pvals, method="holm" | "fdr_bh"); scipy >= 1.11 has stats.false_discovery_control for BH/BY.

Gotchas

• Peeking: testing repeatedly as data arrive and stopping at p < 0.05 inflates false positives far above 5%. Fix the sample size or use sequential methods (see experiment-analysis).
• Subgroup fishing after a null overall result is exploratory; label it and correct.
• Statistical significance with huge n can be practically meaningless; lead with the effect size.
• “p = 0.06, trending toward significance” is not a finding. Report the CI.
• Ratios of means across units (revenue per session when randomized by user) need the delta method or bootstrap by unit, not a t-test on sessions.
• Excluding outliers after seeing results is a forking path; set exclusion rules first.

Output: result statement

**Question:** Does the new onboarding reduce time-to-first-project? (two-sided, alpha 0.05, pre-registered)
**Data:** 412 new users (control 205, new 207), one observation per user; times right-skewed.
**Test:** Mann-Whitney U (primary, declared), Welch t on log time (sensitivity).
**Result:** median 34 min vs 27 min; Hodges-Lehmann shift -6.2 min (95% CI -10.1 to -2.4), p = 0.002;
rank-biserial r = 0.18. Sensitivity analysis agrees.
**Multiple comparisons:** 3 secondary outcomes, Holm-adjusted; none significant.
**Interpretation:** New onboarding likely shortens time-to-first-project by about 2-10 minutes.
**Limits:** single 3-week cohort; desktop only.

Verification

• Test choice matches the decision table row for outcome type, groups, and pairing; independence unit stated.
• Effect size and CI reported in original units next to every p-value.
• Family size counted and the correction applied; adjusted values shown.
• A sensitivity check (alternative test, bootstrap, or excluding outliers by the pre-set rule) agrees in direction; if not, report the disagreement.