Survey Design skill for Claude Code
Questions that map to decisions, samples that hold up. Claude maps every question to a decision before writing it, rewrites double-barreled, leading and vague items, picks fully labeled scales, and orders the flow so data quality holds. A bundled calculator sizes the sample from the smallest subgroup you need, with finite-population correction and design effect, and the analysis plan and exclusion rules are written before fielding.
- name
survey-design - size167 lines · ≈2,050 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Designs surveys that produce decision-grade data — research objectives mapped to questions, neutral wording, balanced response scales, logical flow and skip logic, sampling and sample-size math with margin of error, cognitive pretesting and soft launch, and a pre-registered analysis plan with data-quality screens. Use when writing or reviewing a questionnaire, customer or employee survey, NPS or satisfaction survey, Likert items, market or academic survey, or when asking how many responses are needed or why survey results look biased.Try asking
- Review this customer survey draft and fix any biased or confusing questions
- How many responses do I need from our 12,000 users to compare free and paid plans?
- Design a 10-minute employee engagement survey with an analysis plan
Use it when
- You are writing or reviewing a questionnaire
- You need sample size or margin of error for a survey
- Survey results look skewed and you suspect the instrument
- You want an analysis plan before launching
Skip it when
- You are analyzing an A/B test or product experiment — use experiment-analysis
- You already have open-ended answers to code into themes — use qualitative-coding
How to check the result
Every question appears in the question map, the sample-size output supports the smallest reported subgroup, and the soft launch shows no broken skip paths or high-drop-off items.
The common mistake it prevents
Reporting a self-selected website or social survey with a margin of error as if it represented all customers.
--- name: survey-design description: Designs surveys that produce decision-grade data — research objectives mapped to questions, neutral wording, balanced response scales,… license: MIT ---
Done well means: every question traces to a decision or hypothesis, no question leads, doubles up, or assumes; scales are balanced and consistent; the sample can support the comparisons planned; the instrument was pretested on real respondents; and the analysis plan existed before the data did.
Hard rules
- No question without a named use in the analysis plan. Cut “nice to know” items; every extra minute costs completion rate and data quality.
- Never report a survey of a convenience sample (social media, website pop-up) as representative of a population. State the frame and its coverage gaps.
- Do not collect personal data you will not analyze. Name the legal basis and retention for any identifiable data (GDPR/CCPA), and get ethics approval (IRB) for academic human-subjects research.
- Pretest before fielding. A survey that skipped piloting is a pilot.
Workflow
- [ ] 1. Objectives -> decisions -> hypotheses -> constructs
- [ ] 2. Population, frame, sampling method, sample size
- [ ] 3. Draft questions (wording rules below)
- [ ] 4. Choose scales and response options
- [ ] 5. Order, flow, skip logic, length budget
- [ ] 6. Analysis plan and data-quality screens (before fielding)
- [ ] 7. Cognitive interviews (5-10) and soft launch (5-10% of sample)
- [ ] 8. Field, monitor, clean, analyze per plan1. Objectives to constructs
Build a question map before writing any question:
| Decision | Hypothesis | Construct | Question IDs | Analysis |
|----------|-----------|-----------|--------------|----------|
| Build offline mode? | Field users lose work weekly | Frequency of connectivity loss | Q4, Q5 | % weekly+, by role (Q2) |Use validated instruments when the construct has one (SUS for usability, UMUX-Lite, standardized scales in psychology/health). Do not edit validated items; that breaks comparability.
2. Sampling and size
| Method | Use when | Caveat |
|---|---|---|
| Simple random from a full list | You have a complete frame (customer DB, employee list) | Check list coverage (inactive users, missing emails) |
| Stratified random | Need precise subgroup estimates | Weight back to population proportions |
| Panel / quota | No frame; general population | Non-probability: report as such, avoid MoE claims or label them “credibility intervals” |
| Intercept / convenience | Exploratory, directional only | Self-selection bias; never generalize |
Compute size from the smallest subgroup you need to report, not the total:
python3 scripts/sample_size.py n --moe 0.05 --conf 0.95 --population 12000 --response-rate 0.25
python3 scripts/sample_size.py moe --completes 400 # also prints subgroup MoEs
python3 scripts/sample_size.py diff --p1 0.30 --p2 0.38 --power 0.8 # per group to detect a gapRules of thumb: about 385 completes gives +/-5% at 95% confidence for a proportion in a
large population; about 100 per subgroup gives roughly +/-10%. Use --deff > 1 for
clustered or heavily weighted samples. Response rates: in-product 5-15%, customer email
10-30%, employee 30-70% (highly variable; use your own history).
3. Wording rules
| Problem | Bad | Fix |
|---|---|---|
| Double-barreled | “How satisfied are you with the speed and reliability?” | Split into two items |
| Leading | “How much do you love the new dashboard?” | “How would you rate the new dashboard?” |
| Loaded / assumes | “What do you dislike about our pricing?” | Filter first: “Have you looked at our pricing?” |
| Vague quantifier | “Do you often…?” | “In the past 7 days, how many times…?” |
| Jargon | “How useful is SSO provisioning?” | Plain words, or define the term |
| Negation | “Do you agree the app is not hard to use?” | Positive, direct statement |
| Hypothetical WTP | “Would you pay $20?” | Prefer past behavior, or Van Westendorp / conjoint with caveats |
| Recall overload | “In the past year, how many…” | Shorter reference period matched to frequency |
- Ask about specific, recent behavior before opinions.
- Keep sentences under ~20 words; one idea per item.
- Provide “Not applicable” and “Don’t know” where they are genuinely possible, but not as an easy escape on core items.
- Sensitive topics: ranges instead of exact values (income), indirect wording, late in the survey, explicit confidentiality statement.
4. Scales
| Measure | Default scale |
|---|---|
| Agreement | 5-point: Strongly disagree … Strongly agree, all points labeled |
| Satisfaction | 5-point fully labeled, bipolar with neutral midpoint |
| Frequency | Concrete categories: Never / Less than monthly / Monthly / Weekly / Daily |
| Likelihood to recommend | 0-10 NPS item (report the distribution, not only the score) |
| Importance vs. performance | Paired 5-point items, or MaxDiff for forced tradeoffs |
| Ranking | Rank top 3 only; full ranking of >5 items degrades quality |
- Label every point, not just the ends; keep direction consistent (negative to positive left to right) across the survey.
- Mutually exclusive, exhaustive categories for multiple choice: “18-24, 25-34”, not “18-25, 25-35”; include “Other (specify)”.
- Randomize option order for unordered lists; never randomize ordinal scales.
- Avoid large grids (> 5 rows): they cause straight-lining, especially on mobile.
5. Flow and length
- Order: screeners -> easy behavioral questions -> core topic (general before specific) -> sensitive items -> demographics last (unless needed for quotas or routing).
- Put the most important questions in the first half; quality drops with length.
- Length budget: under 10 minutes for customers (about 20-30 simple items); show progress.
- Test every skip-logic path; a broken branch silently empties a whole section.
- Place an attention check only if the sample is a paid panel; use one instructed-response item, not trick questions.
6. Analysis plan (write before fielding)
Primary questions and the exact metric for each (e.g., % "weekly or more" on Q4)
Planned comparisons (segment A vs. B) and the test (chi-square, two-proportion z, Mann-Whitney for ordinal)
Weighting scheme (variables and population targets) or "unweighted, non-probability"
Exclusion rules: speeders (< 1/3 median completion time), straight-liners on grids, failed attention check, duplicates
Handling of "Don't know"/N/A (excluded from base, reported separately)
Open-ended coding approach (codebook; see qualitative-coding)
Multiple-comparison control if many subgroup tests (Holm or Benjamini-Hochberg)Treat Likert items as ordinal: report distributions and top-2-box; use means only for multi-item scales with known reliability (Cronbach’s alpha >= 0.7).
7. Pretest
- Cognitive interviews (5-10 people from the target population): think aloud, then probe “What does ‘reliable’ mean to you here?” Fix items people interpret differently.
- Soft launch to 5-10% of the sample; check completion time, drop-off by question, item non-response, distribution oddities (everyone in one option), and skip logic. Freeze the instrument before full launch.
Verification
- Every question appears in the question map; every analysis-plan metric has a question.
- Wording table check passes for all items; scales are labeled and consistent.
- Sample-size output supports the smallest reported subgroup at the stated precision.
- Soft-launch data: median time within budget, no question with > 10% drop-off, all routes reached.
Output contract
Deliver: the question map, the instrument (numbered items with response options, logic
notes in brackets like [ASK IF Q3 = Yes]), the sampling plan with size math, and the
analysis plan.
Gotchas
- Changing a question mid-field splits the data into two incomparable groups.
- Incentives raise response rates but attract speeders; screen harder.
- NPS alone does not explain anything; pair it with “What is the main reason for your score?” and code the answers.
- Mobile respondents are often the majority; preview every page on a phone.
- Low response rate is not automatically bias, but compare respondents to the frame on known variables (tenure, plan, region) and weight or caveat accordingly.
Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/sample_size.py
#!/usr/bin/env python3
"""Survey sample-size and margin-of-error calculator (standard library only).
Modes:
n required completes for a target margin of error on a proportion
moe margin of error achieved by a given number of completes
diff completes per group to detect a difference between two proportions
Examples:
python3 sample_size.py n --moe 0.05 --conf 0.95 --population 12000 --response-rate 0.25
python3 sample_size.py moe --completes 400 --population 12000
python3 sample_size.py diff --p1 0.30 --p2 0.38 --power 0.8 --conf 0.95
Options: --p (expected proportion, default 0.5 = most conservative),
--deff (design effect for clustered/weighted samples, default 1.0)
Exit status: 0 ok, 2 invalid input.
"""
from __future__ import annotations
import argparse
import math
import sys
from statistics import NormalDist
def z(conf: float) -> float:
return NormalDist().inv_cdf(1 - (1 - conf) / 2)
def n_for_moe(moe: float, conf: float, p: float, population: int | None, deff: float) -> int:
n0 = (z(conf) ** 2) * p * (1 - p) / moe**2 * deff
if population:
n0 = n0 / (1 + (n0 - 1) / population)
return math.ceil(n0)
def moe_for_n(n: int, conf: float, p: float, population: int | None, deff: float) -> float:
fpc = math.sqrt((population - n) / (population - 1)) if population and population > n else 1.0
if population and n >= population:
return 0.0
return z(conf) * math.sqrt(deff * p * (1 - p) / n) * fpc
def n_per_group_diff(p1: float, p2: float, conf: float, power: float, deff: float) -> int:
za, zb = z(conf), NormalDist().inv_cdf(power)
pbar = (p1 + p2) / 2
num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
return math.ceil(num / (p1 - p2) ** 2 * deff)
def frac(name: str, v: float, lo: float = 0.0, hi: float = 1.0) -> None:
if not (lo < v < hi):
raise ValueError(f"{name} must be between {lo} and {hi} (exclusive), got {v}")
def main(argv: list[str]) -> int:
ap = argparse.ArgumentParser(description="Survey sample size / margin of error")
ap.add_argument("mode", choices=["n", "moe", "diff"])
ap.add_argument("--moe", type=float, help="target margin of error, e.g. 0.05")
ap.add_argument("--completes", type=int, help="number of completed responses")
ap.add_argument("--conf", type=float, default=0.95)
ap.add_argument("--p", type=float, default=0.5, help="expected proportion")
ap.add_argument("--population", type=int, help="population size for finite correction")
ap.add_argument("--response-rate", type=float, help="expected response rate to compute invitations")
ap.add_argument("--deff", type=float, default=1.0, help="design effect")
ap.add_argument("--p1", type=float)
ap.add_argument("--p2", type=float)
ap.add_argument("--power", type=float, default=0.8)
a = ap.parse_args(argv)
try:
frac("conf", a.conf)
frac("p", a.p)
if a.deff < 1:
raise ValueError("deff must be >= 1")
if a.population is not None and a.population < 2:
raise ValueError("population must be >= 2")
if a.mode == "n":
if a.moe is None:
raise ValueError("--moe is required for mode n")
frac("moe", a.moe, 0, 0.5)
n = n_for_moe(a.moe, a.conf, a.p, a.population, a.deff)
print(f"Completes needed: {n} (MoE +/-{a.moe:.1%} at {a.conf:.0%} confidence, p={a.p}, deff={a.deff}"
+ (f", population={a.population}" if a.population else "") + ")")
if a.response_rate:
frac("response-rate", a.response_rate)
print(f"Invitations needed at {a.response_rate:.0%} response: {math.ceil(n / a.response_rate)}")
if a.population and n > 0.5 * a.population:
print("Note: more than half the population is required; consider a census.")
elif a.mode == "moe":
if not a.completes or a.completes < 1:
raise ValueError("--completes (>= 1) is required for mode moe")
m = moe_for_n(a.completes, a.conf, a.p, a.population, a.deff)
print(f"Margin of error: +/-{m:.2%} (n={a.completes}, {a.conf:.0%} confidence, p={a.p}, deff={a.deff})")
for sub in (2, 4):
ns = a.completes // sub
if ns >= 1:
print(f" subgroup of {ns} (1/{sub} of sample): +/-{moe_for_n(ns, a.conf, a.p, None, a.deff):.2%}")
else:
if a.p1 is None or a.p2 is None:
raise ValueError("--p1 and --p2 are required for mode diff")
frac("p1", a.p1)
frac("p2", a.p2)
frac("power", a.power)
if a.p1 == a.p2:
raise ValueError("p1 and p2 must differ")
n = n_per_group_diff(a.p1, a.p2, a.conf, a.power, a.deff)
print(f"Completes per group: {n} (detect {a.p1:.0%} vs {a.p2:.0%}, power {a.power:.0%}, "
f"two-sided alpha {1 - a.conf:.2f}, deff={a.deff}); total {2 * n}")
except ValueError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.