ab-test-setup

Overview of ab-test-setup skill

What ab-test-setup is for

The ab-test-setup skill helps you turn a vague experiment idea into a test plan that is actually runnable for Conversion work. It is best for marketers, growth teams, product marketers, and PMs who need to decide what to test, how to structure it, and whether they have enough traffic to learn anything.

Who should install this skill

Install ab-test-setup if you regularly ask for help with:

• headline or CTA experiments
• landing page and signup flow tests
• variant planning for messaging or offer changes
• sample size, duration, and significance questions
• deciding whether an idea should be A/B tested at all

It is especially useful if your team already has ideas but lacks a repeatable experiment brief.

The real job-to-be-done

Most failed tests do not fail because variant ideas are bad. They fail because the setup is weak: no clear hypothesis, too many changes at once, no baseline, no detectable effect target, or no guardrails. The ab-test-setup skill is designed to force that missing discipline before launch.

What makes this skill different from a generic prompt

A generic prompt will often suggest test ideas. ab-test-setup pushes toward a more valid experiment plan:

• starts from hypothesis, not just “try two versions”
• asks for baseline conversion rate and traffic
• accounts for sample size and test duration
• distinguishes A/B vs A/B/n vs multivariate choices
• warns against peeking and underpowered tests
• points to templates and a sample-size reference in the repo

Best-fit and misfit cases

Best fit:

• you already know the page, audience, and goal
• you need a structured test brief fast
• you want better prompts for Conversion experimentation

Misfit:

• you first need instrumentation or event tracking design
• you want page rewrite ideas without a testing plan
• you have very low traffic and need alternatives to formal testing

How to Use ab-test-setup skill

Install ab-test-setup in your skills environment

Use the repository install pattern shown by the directory baseline:

npx skills add https://github.com/coreyhaines31/marketingskills --skill ab-test-setup

After install, open:

• skills/ab-test-setup/SKILL.md
• skills/ab-test-setup/references/sample-size-guide.md
• skills/ab-test-setup/references/test-templates.md
• skills/ab-test-setup/evals/evals.json

Those files matter more than a quick skim because they show the intended decision flow, output shape, and quality bar.

Read these files first

If you only read three files before using ab-test-setup, read:

1. SKILL.md for trigger conditions and planning logic
2. references/sample-size-guide.md for feasibility and duration decisions
3. references/test-templates.md for the final structure you want the model to produce

Then check evals/evals.json to see what the skill considers a good answer in realistic prompts.

What input ab-test-setup needs

The skill gets much better when you provide:

• page or feature being tested
• primary conversion event
• current baseline conversion rate
• monthly or weekly traffic volume
• proposed change
• audience segment
• tooling constraints
• timeline or launch window
• risk tolerance for false positives

Without baseline and traffic, ab-test-setup usage becomes more generic and less decision-useful.

Start with product marketing context if available

The repo explicitly tells the skill to check .agents/product-marketing-context.md or .claude/product-marketing-context.md first. That matters because good experiment design depends on:

• audience
• positioning
• core claims
• current messaging strategy
• funnel stage

If your environment has that file, make sure the model reads it before asking repetitive discovery questions.

Turn a rough idea into a strong ab-test-setup prompt

Weak prompt:

We want to test our homepage headline. What should we do?

Better prompt:

Use ab-test-setup to plan an A/B test for our homepage headline. Current headline: "The All-in-One Project Management Tool." Proposed direction: more benefit-focused messaging for SaaS team leads. Baseline signup rate is 3.2%. We get about 15,000 homepage visitors per month. Primary goal is signup rate. We can implement one variant only, 50/50 traffic split, in our existing testing tool. Please create a hypothesis, recommend test type, estimate sample needs and likely duration, define primary/secondary/guardrail metrics, and flag risks like peeking or low power.

That second version gives the skill enough context to produce a plan instead of generic brainstorming.

Ask for the output format you actually need

The references include reusable templates, so ask for one of these formats:

• experiment brief for approval
• launch checklist
• test plan template
• stakeholder update
• post-test readout shell

Practical prompt:

Use the test plan template format from references/test-templates.md and fill only fields we can support with the data provided. Mark missing assumptions clearly.

This reduces cleanup work and exposes missing inputs early.

Use the skill for decisions, not just idea generation

The most useful ab-test-setup guide workflow is:

1. describe the proposed change
2. state the business goal
3. provide baseline and traffic
4. ask whether the test is viable
5. ask for exact metrics and run conditions
6. only then ask for variant recommendations

This order matters. It stops teams from over-investing in tests that cannot reach adequate sample size.

Know the core planning rules it enforces

From the source, the skill strongly centers on:

• start with a clear hypothesis
• test one thing at a time
• define primary, secondary, and guardrail metrics
• estimate sample size and minimum duration
• avoid ending tests early based on noisy early wins

If your organization often launches “quick tests” without these controls, this skill adds real value.

How to use ab-test-setup for Conversion work

For ab-test-setup for Conversion, include the business stakes, not just the variant idea. Good inputs:

• current conversion bottleneck
• why the current page may underperform
• expected mechanism of change
• minimum lift worth acting on
• segments that must not degrade

Example:

We think our pricing page CTA underperforms because it asks for commitment too early. Plan an A/B test comparing "Start Free Trial" vs "See Plans First." Baseline click-through is 6.8%, downstream trial-start rate is 2.1%, and pricing page traffic is 40,000 sessions/month. We care most about completed trial starts, not just button clicks. Include guardrails so a CTR lift does not hide lower-quality signups.

That prompt leads to better metric selection than simply asking for a button-color test.

When the skill will push back on your idea

Expect ab-test-setup to be most helpful when it says:

• this should not be multivariate
• you do not have enough traffic for four variants
• your MDE is unrealistically small
• your primary metric is too far from the tested change
• you are mixing too many changes to learn causally

That pushback is a feature, not friction.

Common repo-backed use cases

Based on the skill text and evals, good uses include:

• homepage headline A/B tests
• CTA variant tests on pricing or signup pages
• deciding if A/B/n is realistic
• duration planning from traffic and baseline
• creating structured documentation for experiment rollout

The evals also show the skill should catch casual requests like “should we test 4 CTA colors?” and steer users toward stronger experiment design.

ab-test-setup skill FAQ

Is ab-test-setup good for beginners?

Yes, if you already understand your page and goal. The skill gives structure beginners often miss: hypothesis, sample size thinking, metrics, and duration. It is less suitable if you need a statistics primer from scratch.

What is the main advantage over ordinary prompting?

The main advantage is constraint. ab-test-setup does not just generate variants; it frames whether the test is worth running and what valid measurement requires. That usually saves more time than idea generation.

Do I need exact traffic and conversion data?

Exact is best, directional is still useful. If you only have rough estimates, say so explicitly. The skill can still produce a planning draft, but confidence in sample-size and duration guidance will be lower.

Can ab-test-setup handle more than two variants?

Yes, but it should also warn that extra variants increase sample requirements. If traffic is modest, an A/B test is often more practical than A/B/n or multivariate testing.

When should I not use ab-test-setup?

Do not use it as your main tool when:

• tracking is missing or unreliable
• traffic is too low for meaningful inference
• you need a CRO rewrite, not a test plan
• the change is so large that implementation feasibility is the real blocker
• you need analytics instrumentation design first

Is this skill tied to one testing platform?

No evidence suggests a platform lock-in. The skill is planning-oriented, so it should work with most experimentation tools as long as you can specify traffic split, metrics, and implementation constraints.

Does ab-test-setup help with post-test analysis?

Partly. The templates include results documentation, but the strongest value is still pre-launch setup. Use it to define what success means before the test starts.

How to Improve ab-test-setup skill

Give stronger hypotheses, not just variant requests

Bad input:

Test this new copy against the old copy.

Better input:

Because users may not understand our current value proposition quickly, we believe replacing feature-led copy with outcome-led copy will increase signup starts among first-time visitors. We will measure signup rate as the primary metric and bounce rate plus demo-request rate as secondary checks.

This gives ab-test-setup a causal story to test, not just two artifacts to compare.

Provide the minimum viable experiment data set

To improve ab-test-setup output quality, always try to include:

• baseline conversion rate
• traffic volume
• minimum meaningful lift
• exact conversion event
• audience
• implementation constraints
• acceptable test duration

These inputs directly improve sample-size logic and feasibility recommendations.

Avoid the most common failure modes

Weak outputs usually come from one of these:

• too many changes bundled into one test
• no baseline metric
• vanity metric as primary KPI
• asking for significance without traffic reality
• testing an upstream micro-metric while the real business goal is downstream

If you fix those before prompting, the skill becomes much more useful.

Tell the skill what must not get worse

A stronger ab-test-setup skill prompt includes guardrails such as:

• lead quality
• refund rate
• bounce rate
• activation rate
• revenue per visitor

This prevents false “wins” where the top-line metric rises but business quality falls.

Use the sample-size reference as a feasibility filter

Before spending time on variants, check references/sample-size-guide.md. It helps answer:

• can this test finish in a reasonable window?
• is the desired lift too small to detect?
• would fewer variants be smarter?
• should we use a larger change instead of a subtle tweak?

This is one of the highest-value files in the repo for install decisions.

Reuse the templates instead of freeform outputs

references/test-templates.md is the fastest path to better team adoption. Ask the model to fill:

• test plan
• prioritization scorecard
• stakeholder update
• hypothesis bank entry

Freeform responses are easy to generate but harder to operationalize.

Iterate after the first draft

After the first ab-test-setup usage pass, do one refinement round:

1. tighten the hypothesis
2. cut scope to one variable
3. replace weak metrics with operational definitions
4. confirm traffic split and duration
5. ask what assumptions are still missing

That second pass often improves the plan more than adding more variant ideas.

Pair ab-test-setup with adjacent skills carefully

The skill itself points to adjacent needs:

• use analytics-tracking if measurement setup is the blocker
• use page-cro if you need page-level optimization ideas before formal testing

That division is useful. ab-test-setup is strongest once you already know what change you want to evaluate and need a valid experiment plan.