Conversion Optimization skill for Claude Code
Research-led A/B tests sized before they start. Claude runs CRO as a research process, not a list of button tweaks. It quantifies funnel leaks by device, pairs each with qualitative evidence from replays and surveys, and writes falsifiable hypotheses ranked with ICE. A bundled stdlib calculator sizes tests, finds the detectable effect for your traffic, and flags sample ratio mismatch before anyone reads the results.
- name
conversion-optimization - size178 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Runs a conversion rate optimization program from funnel analysis to shipped experiments, covering drop-off diagnosis, qualitative research (session replays, surveys, user tests), evidence-backed hypotheses, ICE prioritization, A/B test design with sample size, run length, guardrails, and SRM checks, and a test log. Use when signup, checkout, or trial conversion is low, when planning A/B tests in PostHog, Optimizely, VWO, GrowthBook, or Statsig, when asked how long a test must run, or when deciding which page changes to try first.Try asking
- Our checkout converts at 1.8% on mobile vs 3.9% on desktop. Find out why and plan tests
- How long do we need to run a test on a 4% signup rate to detect a 10% lift?
- Prioritize these 12 test ideas from the team and design the first experiment
Use it when
- Signup, checkout, or trial conversion is below expectations
- Planning or sizing an A/B test
- Turning a backlog of test ideas into a prioritized roadmap
- A test result looks suspicious or the traffic split is uneven
Skip it when
- You need rigorous post-test statistics like CUPED or sequential analysis — use experiment-analysis
- You only need the page copy rewritten — use landing-page-copy
How to check the result
Each test has a precomputed sample size and run length from ab_calc.py, passes the SRM check at readout, and is logged with its primary metric, guardrails, and decision.
The common mistake it prevents
Stopping a test as soon as the dashboard shows significance, which turns random noise into a shipped 'winner' that never holds up.
--- name: conversion-optimization description: Runs a conversion rate optimization program from funnel analysis to shipped experiments, covering drop-off diagnosis, qualitative research… license: MIT ---
CRO done well finds where and why qualified visitors drop, tests the changes most likely to move the business metric, and only declares winners from tests that were sized in advance, ran full weeks, passed a sample-ratio check, and did not hurt guardrails. Most of the value comes from research and prioritization, not from button colors.
Ground rules
- Diagnose before prescribing. No hypothesis without a quantitative signal (where) and a qualitative one (why).
- Optimize for the metric that pays: completed purchases, activated accounts, retained revenue — not clicks on the CTA you changed.
- Size every test before launch. If the page lacks traffic to detect a realistic effect, do not A/B test it; ship research-backed fixes and monitor, or test higher in the funnel.
- Fix bugs and broken flows directly; they are not experiments.
- Never stop a test the moment it “reaches significance” (peeking inflates false positives) unless the tool uses sequential testing designed for it.
- Respect consent and privacy: session replay and heatmap tools must mask inputs and load only under the consent the user’s region requires.
Workflow
- [ ] 1. Define the conversion and baseline
- [ ] 2. Map the funnel and find the biggest leaks
- [ ] 3. Research why (qualitative)
- [ ] 4. Write hypotheses
- [ ] 5. Prioritize with ICE
- [ ] 6. Design and size the test
- [ ] 7. QA and launch
- [ ] 8. Analyze, decide, log1. Conversion and baseline
Name the primary conversion event, its unit (visitor, session, account), and the baseline
rate over the last 4–8 full weeks, split by device and top traffic sources. Confirm the
event fires once per conversion and matches backend records within ~5%; if not, fix
tracking first (analytics-tracking-plan).
2. Funnel and leaks
Build the step funnel in the analytics tool (GA4 funnel exploration, PostHog/Mixpanel/ Amplitude funnels), per device:
| Step | Users | Step conversion | Drop-off | Mobile vs desktop |
|---|---|---|---|---|
| Landing view | 40,000 | — | — | |
| Pricing view | 9,200 | 23% | 77% | |
| Signup start | 2,300 | 25% | 75% | |
| Signup complete | 1,150 | 50% | 50% | mobile 31% / desktop 62% |
Leak size = users lost × value per conversion. Big device gaps, steps below peer pages,
and error-event spikes are the leads. Check page speed for the leaking step (field LCP and
INP at p75) — slow mobile steps are a common hidden cause; hand off to core-web-vitals.
3. Qualitative research
| Method | Answers | Volume needed |
|---|---|---|
| Session replays of drop-offs | Where people hesitate, rage-click, hit errors | 20–40 sessions per leaking step |
| On-page exit or intent survey (1 question) | “What’s stopping you from signing up today?” | 50–200 responses |
| Post-conversion survey | What almost stopped buyers; what convinced them | 50+ |
| 5 moderated user tests | Comprehension, trust, usability failures | 5 per segment finds most major issues |
| Sales/support transcripts | Objections in customer words | All recent |
| Form analytics | Field-level abandonment and errors | Per form |
Code findings into themes with counts. A theme seen in both data and research becomes a hypothesis.
4. Hypotheses
Format every hypothesis so it can be wrong:
Because we saw <evidence: data + research>, we believe that <change> for <segment>
will cause <metric> to <direction>. We'll know when <primary metric> moves by ≥ <MDE>
without <guardrail> worsening.Prefer changes that address the motivation, clarity, trust, or friction problem the research revealed: clearer value proposition, answering an objection near the CTA, removing form fields, showing price earlier, adding guest checkout, fixing mobile layout.
5. Prioritize with ICE
Score 1–10, average or multiply, sort descending:
| Factor | High score means |
|---|---|
| Impact | Large leak × strong evidence × close to revenue |
| Confidence | Supported by both quant and qual data, or proven in similar tests |
| Ease | Days not weeks to build, no cross-team dependencies, enough traffic to test |
Break ties toward bolder changes: on low-to-mid traffic, small tweaks cannot reach significance. If a page needs > 8 weeks to detect the expected effect, it scores low on Ease.
6. Design and size the test
Use the bundled calculator (two-sided z-test for proportions):
python3 scripts/ab_calc.py size --baseline 0.042 --mde 0.10 --daily 3000
python3 scripts/ab_calc.py mde --baseline 0.042 --per-variant 20000Design rules:
- MDE: the smallest lift worth acting on, usually 5–20% relative. Smaller MDE → quadratically more traffic.
- Defaults: alpha 0.05 two-sided, power 0.8, 50/50 split. With multiple variants the script applies Bonferroni; prefer A/B over A/B/C/D unless traffic is plentiful.
- Run length: whole weeks (minimum 1, ideally 2+) to cover weekday cycles; stop at the precomputed sample size, not at first significance. Cap at ~4–6 weeks; cookie churn and seasonality degrade longer tests.
- Randomization unit: user (logged-in ID or stable device ID), not pageview, so a person always sees one variant.
- Primary metric: one, decided in advance. Guardrails: revenue per visitor, refund rate, error rate, page performance, downstream activation.
- Targeting: only the users who reach the changed experience (trigger-based exposure logging) — diluting with unexposed users hides effects.
- Flicker: client-side testing tools can flash the original; use server-side or edge assignment or the tool’s anti-flicker snippet with a short timeout.
7. QA and launch
- Both variants render on target devices and browsers; screenshots attached.
- Exposure event fires once per user with the variant name; conversion events attribute to it.
- Internal traffic and bots excluded.
- No other test overlaps the same users and metric (or tests are mutually exclusive).
- Rollback is one switch (feature flag), documented.
8. Analyze, decide, log
- SRM check first:
python3 scripts/ab_calc.py srm <control_n> <variant_n>. If it exits 1 (p < 0.001), the assignment or logging is broken; discard results until fixed. - Read the primary metric with its confidence interval, not just the p-value. Then guardrails. Segment cuts (device, new vs. returning) are exploratory; do not declare segment winners unless pre-registered.
- Decide: ship (primary up, guardrails flat), iterate (directional, learnings clear), or kill. Inconclusive is a valid result — do not run “until it wins”.
- Hand deep statistical questions (CUPED, sequential testing, Bayesian readouts) to
experiment-analysis. - Log every test, including losers, in the template below.
Gotchas
- Novelty effects: a redesign often spikes in week 1 and decays. Look at the trend over the full run.
- Testing on pages where most traffic is returning customers measures habit disruption, not persuasion.
- Measuring “click on CTA” as primary rewards clickbait CTAs that lose at checkout.
- Multiple simultaneous changes in one variant are fine for big-swing tests, but you will not know which change worked; follow up with isolating tests if it matters.
- Before/after comparisons are not tests; seasonality, campaigns, and releases confound them.
- Heatmaps aggregate across devices and scroll depths; segment them or they mislead.
Output: test log entry
### T-014 Pricing page: show annual price per month + objection FAQ near CTA
Hypothesis: Because 38% of exit-survey answers cite "unclear what I'd pay" and replays show
pricing → FAQ scroll loops, showing monthly-equivalent price and a 4-question FAQ beside the
CTA will raise trial starts from pricing viewers by ≥ 10%.
ICE: I8 C7 E8 = 7.7 · Primary: trial_started / pricing viewers · Guardrails: paid conversion at 30d, RPV
Design: 50/50 user-level, baseline 4.2%, MDE 10% rel, n = 39,000/variant, 21 days
SRM: p = 0.41 (pass)
Result: +11.8% (95% CI +3.1% to +20.5%), guardrails flat → Ship
Learning: Price clarity outweighs discount framing for SMB visitors.Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/ab_calc.py
#!/usr/bin/env python3
"""A/B test planning and sanity checks (stdlib only).
Sample size for a conversion-rate test (two-sided, per variant):
python3 ab_calc.py size --baseline 0.042 --mde 0.10 [--alpha 0.05] [--power 0.8]
[--variants 2] [--daily 3000]
--mde is RELATIVE lift (0.10 = +10%). Use --abs-mde for absolute points (0.004).
--daily = eligible visitors per day across all variants; prints run length.
Minimum detectable effect for a fixed traffic budget:
python3 ab_calc.py mde --baseline 0.042 --per-variant 20000
Sample ratio mismatch check (chi-square goodness of fit vs. intended split):
python3 ab_calc.py srm 50312 49120 [--weights 50 50]
Exits 1 when p < 0.001 (treat results as untrustworthy until explained).
"""
from __future__ import annotations
import argparse
import math
import sys
from statistics import NormalDist
Z = NormalDist()
def n_per_variant(p1: float, p2: float, alpha: float, power: float, comparisons: int) -> int:
a = alpha / comparisons # Bonferroni for multiple treatment arms vs. control
za = Z.inv_cdf(1 - a / 2)
zb = Z.inv_cdf(power)
pbar = (p1 + p2) / 2
num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
return math.ceil(num / (p2 - p1) ** 2)
def gammaincc(s: float, x: float) -> float:
"""Regularized upper incomplete gamma Q(s, x)."""
if x <= 0:
return 1.0
if x < s + 1: # series for P, return 1 - P
term = total = 1.0 / s
k = s
for _ in range(500):
k += 1
term *= x / k
total += term
if abs(term) < abs(total) * 1e-15:
break
return 1.0 - total * math.exp(-x + s * math.log(x) - math.lgamma(s))
# continued fraction (Lentz) for Q
b = x + 1 - s
c = 1 / 1e-300
d = 1 / b
h = d
for i in range(1, 500):
an = -i * (i - s)
b += 2
d = an * d + b
d = 1e-300 if abs(d) < 1e-300 else d
c = b + an / c
c = 1e-300 if abs(c) < 1e-300 else c
d = 1 / d
delta = d * c
h *= delta
if abs(delta - 1) < 1e-15:
break
return math.exp(-x + s * math.log(x) - math.lgamma(s)) * h
def cmd_size(a) -> int:
p1 = a.baseline
p2 = p1 + a.abs_mde if a.abs_mde else p1 * (1 + a.mde)
if not 0 < p1 < 1 or not 0 < p2 < 1:
print("ERROR: rates must be between 0 and 1")
return 2
comps = max(1, a.variants - 1)
n = n_per_variant(p1, p2, a.alpha, a.power, comps)
total = n * a.variants
print(f"baseline {p1:.4%} -> target {p2:.4%} (relative {p2 / p1 - 1:+.1%}, absolute {p2 - p1:+.4f})")
print(f"alpha {a.alpha} two-sided{' (Bonferroni over %d comparisons)' % comps if comps > 1 else ''}, power {a.power}")
print(f"sample size: {n:,} per variant, {total:,} total")
if a.daily:
days = math.ceil(total / a.daily)
weeks = math.ceil(days / 7)
print(f"run length at {a.daily:,}/day: {days} days -> run {max(weeks, 1) * 7} days (whole weeks, minimum 7)")
if days > 56:
print("WARNING: > 8 weeks. Test a bolder change, a higher-traffic page, or a higher-rate metric.")
return 0
def cmd_mde(a) -> int:
p1, n = a.baseline, a.per_variant
lo, hi = 1e-6, 1 - p1 - 1e-9
for _ in range(100): # bisection on absolute lift
mid = (lo + hi) / 2
if n_per_variant(p1, p1 + mid, a.alpha, a.power, 1) > n:
lo = mid
else:
hi = mid
print(f"with {n:,} per variant at baseline {p1:.4%}: MDE = +{hi:.4f} absolute ({hi / p1:+.1%} relative)")
return 0
def cmd_srm(a) -> int:
counts = a.counts
weights = a.weights or [1] * len(counts)
if len(weights) != len(counts):
print("ERROR: --weights must match number of counts")
return 2
total, wsum = sum(counts), sum(weights)
chi2 = sum((c - total * w / wsum) ** 2 / (total * w / wsum) for c, w in zip(counts, weights))
df = len(counts) - 1
p = gammaincc(df / 2, chi2 / 2)
shares = ", ".join(f"{c / total:.2%}" for c in counts)
print(f"observed shares: {shares}; chi2 = {chi2:.2f}, df = {df}, p = {p:.6f}")
if p < 0.001:
print("SRM DETECTED: assignment or logging is broken. Do not read the results until fixed.")
return 1
print("No sample ratio mismatch at p < 0.001.")
return 0
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = ap.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("size")
s.add_argument("--baseline", type=float, required=True)
s.add_argument("--mde", type=float, default=0.10)
s.add_argument("--abs-mde", type=float)
s.add_argument("--alpha", type=float, default=0.05)
s.add_argument("--power", type=float, default=0.8)
s.add_argument("--variants", type=int, default=2)
s.add_argument("--daily", type=int)
m = sub.add_parser("mde")
m.add_argument("--baseline", type=float, required=True)
m.add_argument("--per-variant", type=int, required=True)
m.add_argument("--alpha", type=float, default=0.05)
m.add_argument("--power", type=float, default=0.8)
r = sub.add_parser("srm")
r.add_argument("counts", type=int, nargs="+")
r.add_argument("--weights", type=float, nargs="+")
a = ap.parse_args()
return {"size": cmd_size, "mde": cmd_mde, "srm": cmd_srm}[a.cmd](a)
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.