experiment-designer
by alirezarezvaniexperiment-designer helps Product Management teams design A/B and feature experiments with testable hypotheses, metrics, sample-size planning, ICE prioritization, stopping rules, and result interpretation.
This skill scores 80/100, which means it is a solid listing candidate for directory users who want an agent to structure product experiments with more rigor than a generic prompt. It has clear activation cues, an actionable workflow, supporting references, and a practical script, though adoption would be easier with clearer installation guidance and broader statistical tooling.
- Clear trigger scope for product experiment planning, hypothesis writing, sample-size estimation, ICE prioritization, and A/B outcome interpretation.
- Operational workflow gives agents concrete steps: If/Then/Because hypothesis, metric hierarchy, sample-size estimation, ICE scoring, stopping rules, and interpretation.
- Useful progressive disclosure through references for experiment playbook guidance and statistics concepts, plus a runnable sample-size calculator script.
- No install command or README is present in the skill folder, so users must infer installation from the broader repository conventions.
- The included calculator covers two-proportion conversion tests; mean-based tests, sequential methods, and multivariate power planning are not implemented.
Overview of experiment-designer skill
What experiment-designer does
experiment-designer is a Product Management skill for turning product ideas into testable experiments with clearer hypotheses, pre-defined metrics, sample-size planning, ICE prioritization, launch rules, and result interpretation. It is most useful when a team needs more rigor than “let’s A/B test it” but does not want to build a full experimentation playbook from scratch.
Best-fit users and decisions
The experiment-designer skill fits PMs, growth leads, UX researchers, analysts, and startup teams planning A/B tests, multivariate tests, holdouts, or feature experiments. It helps answer practical questions such as: What is the decision metric? What MDE is worth detecting? How much traffic is needed? Which experiment should run first? What would make the result trustworthy enough to ship, roll back, or rerun?
What makes it different from a generic prompt
The repository includes a structured workflow in SKILL.md, a product experimentation playbook in references/experiment-playbook.md, a PM-friendly statistics reference in references/statistics-reference.md, and a Python helper script for two-proportion sample-size estimates. That combination makes experiment-designer more operational than a generic brainstorming prompt: it pushes the agent toward hypothesis quality, guardrail metrics, fixed stopping rules, and practical significance.
Important limits before installing
This is not a full experimentation platform, causal inference library, or analytics SDK. The included scripts/sample_size_calculator.py is aimed at two-proportion A/B tests, so continuous metrics, ratio metrics, sequential testing, clustered assignment, and complex marketplace experiments will need extra statistical review. Use it to improve experiment design quality, not to replace analyst validation for high-stakes decisions.
How to Use experiment-designer skill
experiment-designer install and first files to read
Install the skill from the GitHub repository with:
npx skills add alirezarezvani/claude-skills --skill experiment-designer
After install, read files in this order:
SKILL.mdfor the core workflow and trigger context.references/experiment-playbook.mdfor experiment types, metric design, stopping rules, and pre-launch checks.references/statistics-reference.mdfor p-values, confidence intervals, MDE, power, and practical significance.scripts/sample_size_calculator.pyif you need traffic or runtime estimates for conversion-rate tests.
The skill path is product-team/skills/experiment-designer.
Inputs the skill needs for strong output
For useful experiment-designer usage, provide more than a feature idea. Include the product area, user segment, proposed intervention, current baseline metric, target metric movement, traffic volume, launch constraints, and business risk. The skill works best when it can distinguish the primary decision metric from guardrails and diagnostics.
Weak prompt:
Design an experiment for our onboarding flow.
Stronger prompt:
Use experiment-designer for Product Management. We want to test replacing a 5-step onboarding checklist with a guided setup wizard for new B2B workspace admins. Current activation is 12% within 7 days. We care about activation as the primary metric, but must guardrail support tickets, setup completion time, and day-14 retention. Daily eligible users are about 1,200. We would ship only if uplift is at least 2 absolute percentage points and guardrails do not worsen materially.
Practical workflow for planning an experiment
Start by asking the skill to produce an If/Then/Because hypothesis, not a test plan immediately. Then ask it to define one primary metric, guardrail metrics, diagnostic metrics, and exclusion rules. Next, run or request sample-size planning using the baseline rate, MDE, alpha, power, and daily samples.
For a conversion experiment, the included script can be used like:
python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute --daily-samples 1200
Then use the result to decide whether the experiment is feasible, too underpowered, or should be reframed with a larger expected effect, broader population, or lower-cost learning method.
Prompt pattern that invokes the skill well
Use a prompt shaped like this:
Apply the experiment-designer skill. Create an experiment brief for
[intervention]targeting[segment]. Baseline[primary metric]is[value]; the smallest useful effect is[MDE]; daily eligible traffic is[volume]. Include hypothesis, primary/guardrail/diagnostic metrics, sample-size assumptions, ICE prioritization, stopping rules, launch checklist, and result interpretation guidance for ship/no-ship decisions.
This structure gives the agent the information needed to avoid vague metrics, unrealistic runtimes, and post-hoc decision rules.
experiment-designer skill FAQ
Is experiment-designer only for A/B testing?
No. The experiment-designer skill covers A/B tests, multivariate tests, holdout tests, hypothesis writing, metric selection, prioritization, and interpretation. However, its built-in calculator is specifically for two-proportion conversion experiments, so other designs may need separate methods.
Can beginners use experiment-designer for Product Management?
Yes, especially PMs who need a practical guide to experiment planning without deep statistics training. The statistics reference explains concepts like p-value, confidence interval, MDE, power, Type I/II errors, and practical significance in product language. Beginners should still review plans with an analyst when decisions are expensive or irreversible.
When should I not use this skill?
Do not rely on it alone for sequential testing, network effects, marketplace interference, long-term retention studies, non-randomized causal claims, or experiments with legal, medical, financial, or safety consequences. It can help structure the decision, but it does not guarantee valid inference under complex assumptions.
How is this better than asking for an experiment plan?
A normal prompt may produce a polished plan while missing MDE, stopping rules, guardrails, or feasibility. experiment-designer is opinionated around the failure modes that commonly damage product experiments: changing metrics midstream, peeking too often, underestimating sample size, overvaluing statistical significance, and ignoring implementation cost.
How to Improve experiment-designer skill
Improve experiment-designer inputs before asking for a plan
The fastest way to improve experiment-designer output is to provide real constraints. Add baseline conversion, desired MDE, traffic by segment, expected runtime, rollout limits, instrumentation status, and what decision the result must support. If you do not know these, ask the skill to list assumptions and missing inputs before drafting the experiment brief.
Watch for common failure modes
Check the first output for these issues: multiple “primary” metrics, vague guardrails, no minimum run duration, no randomization notes, unrealistic sample size, success criteria based only on p-value, or diagnostics being treated as ship/no-ship gates. Ask the skill to revise against the checklist in references/experiment-playbook.md.
Iterate from brief to decision memo
A good workflow is: rough idea → experiment brief → sample-size check → feasibility decision → launch checklist → interpretation memo. After results arrive, provide observed effect size, confidence interval, p-value if available, guardrail outcomes, sample size reached, and any data-quality issues. Then ask experiment-designer to separate statistical significance, practical significance, and recommended product action.
Extend the skill for your team
Teams can improve the skill by adding company-specific metric definitions, standard guardrails, experimentation platform conventions, approved alpha/power defaults, segmentation rules, and examples of past good decisions. If your organization runs many experiments on continuous metrics or uses sequential methods, add separate references rather than overloading the existing two-proportion calculator.