ab-test-analysis
by phurynab-test-analysis helps you evaluate A/B test results with statistical rigor, including sample size validation, confidence intervals, significance testing, and ship/extend/stop recommendations. Use it for experiment review, split-test interpretation, and decision-making for Data Analysis workflows.
This skill scores 78/100, which means it is a solid listing candidate for directory users: it clearly targets A/B test analysis, gives enough workflow detail to reduce guesswork, and should be installable for agents that need experiment-readout support, though it is not fully packaged with supporting files or install guidance.
- Explicit trigger language for A/B test analysis, significance checks, sample size validation, and ship/stop recommendations.
- Operational workflow is spelled out with steps for understanding the experiment, validating setup, and calculating statistical significance.
- Body is substantive (3232 chars) with concrete statistical formulas and code-fence usage, which gives agents more executable guidance than a generic prompt.
- No install command or supporting reference files are provided, so adoption may require the user to inspect the SKILL.md directly.
- Experimental/test-like signals appear in the content, and the repository lacks external validation assets, so users should treat it as a focused utility rather than a heavily supported package.
Overview of ab-test-analysis skill
What ab-test-analysis does
The ab-test-analysis skill helps you evaluate experiment results with statistical discipline, then turn the numbers into a practical ship, extend, or stop decision. It is built for users who need more than a quick read of uplift: ab-test-analysis checks whether the test was set up well enough to trust the result, not just whether the variant “won.”
Who it is best for
Use this ab-test-analysis skill if you work in product, growth, analytics, or experimentation and need a repeatable way to review A/B tests. It is a strong fit for ab-test-analysis for Data Analysis when the task is to interpret conversion data, validate significance, and communicate the outcome clearly to non-technical stakeholders.
What problem it solves
The real job-to-be-done is deciding whether a result is actionable. ab-test-analysis helps with sample size validation, confidence intervals, significance testing, and guardrail checks so you can avoid shipping a misleading result or overreacting to noise.
What makes it worth installing
The main value of ab-test-analysis is decision quality. It is designed to read experiment inputs directly, support file-based analysis, and produce recommendations grounded in experiment hygiene such as duration, randomization, and statistical power. If you need an ab-test-analysis guide that is practical rather than theoretical, this skill is a good fit.
How to Use ab-test-analysis skill
Install and locate the skill
Run the ab-test-analysis install flow with the repo command:
npx skills add phuryn/pm-skills --skill ab-test-analysis
After install, open SKILL.md first. In this repository, that file contains the working instructions and is the highest-signal source for the ab-test-analysis usage path.
What to provide in your prompt
The skill works best when you give it the experiment context, not just raw numbers. Include the hypothesis, control and variant definitions, primary metric, guardrails, traffic split, test duration, and any data files you have. A strong prompt looks like:
“Analyze this A/B test for checkout button color. Primary metric is purchase conversion, guardrail is refund rate, traffic split is 50/50, test ran 14 days, and I’m attaching the CSV export. Please check sample size, SRM, confidence interval, and recommend ship/extend/stop.”
Practical workflow
Start with the result file, then confirm the experiment setup, then ask for the statistical readout, and only then ask for the decision. That sequence matters because ab-test-analysis is meant to catch underpowered tests, randomization issues, and timing problems before you accept the uplift.
Files and clues to read first
This repository is compact, so SKILL.md is the key file to inspect first. If the directory expands later, prioritize any README.md, AGENTS.md, metadata.json, rules/, resources/, references/, or scripts/ folders because those are the most likely places for workflow constraints, examples, or calculation helpers.
ab-test-analysis skill FAQ
Is ab-test-analysis only for conversion tests?
No. It is best known for conversion-rate experiments, but the ab-test-analysis skill is also useful whenever you need to compare two variants with a primary metric, confidence intervals, and a decision recommendation. It is less useful when the task is descriptive reporting with no experimental design.
Do I need a statistics background?
Not much. The skill is useful for beginners because it can structure the analysis and explain the result clearly. That said, ab-test-analysis works best when you can provide clean inputs and answer basic questions about the hypothesis, metric, and test design.
How is this different from a normal prompt?
A generic prompt often jumps straight to significance. ab-test-analysis adds a more complete workflow: verify the experiment setup, check sample size and duration, look for SRM or novelty effects, then compute and interpret the result. That extra structure usually leads to better decisions than a one-off analysis prompt.
When should I not use it?
Do not use ab-test-analysis if you only need dashboard narration, marketing copy, or a report with no statistical judgment. It is also a poor fit when the dataset is incomplete and you cannot identify the control, variant, metric, or test window.
How to Improve ab-test-analysis skill
Give the skill better experiment context
The biggest quality gain comes from stronger input. Include the hypothesis, exact change, segment definition, metric formula, duration, and any exclusions. If you omit these, ab-test-analysis may still compute numbers, but the recommendation will be weaker because it cannot judge whether the test design supports the result.
Share data in the most analysis-friendly form
If you have a CSV or export, include one row per unit or the aggregated counts needed for analysis. The skill can read data files directly, so give it the rawest version that still preserves privacy and structure. Avoid screenshots of charts when you can provide tables, because tables make significance and sample checks much more reliable.
Ask for the decision you actually need
The best ab-test-analysis usage is decision-shaped. Instead of asking “Is this significant?”, ask for “ship, extend, or stop with the reasoning and caveats.” That pushes the output toward business action, not just statistical output.
Iterate after the first pass
If the first analysis shows weak power, SRM, or mixed guardrail movement, refine the request with more context rather than forcing a conclusion. Common improvements include adding precomputed counts, clarifying the primary metric, or asking for a breakdown by segment or time window. That is the fastest way to get a better ab-test-analysis guide for your specific experiment.