A B Test Design

Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.

48 stars 10 forksTesting & QA Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add sethdford/claude-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add sethdford/claude-skills --skill a-b-test-design
Manual install
git clone --depth 1 https://github.com/sethdford/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/designer/prototyping-testing/skills/a-b-test-design ~/.claude/skills/a-b-test-design

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
## What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
## Test Structure
### 1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
### 2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
### 3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
### 4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
### 5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
### 6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
## Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
## When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
## Best Practices
- One hypothesis per test
Read the full skill on GitHub

Details

Skill name
a-b-test-design
Path
designer/prototyping-testing/skills/a-b-test-design/SKILL.md
License
MIT
Repo updated
11 March 2026

FAQ

What does the A B Test Design skill do?

Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.

How do I install the A B Test Design skill?

Claude Code plugin: /plugin marketplace add sethdford/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is A B Test Design popular?

It lives in sethdford/claude-skills, which has 48 GitHub stars and 10 forks.

Related skills

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.