A B Test Design
Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.
Install
/plugin marketplace add sethdford/claude-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add sethdford/claude-skills --skill a-b-test-designgit clone --depth 1 https://github.com/sethdford/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/designer/prototyping-testing/skills/a-b-test-design ~/.claude/skills/a-b-test-designPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# A/B Test Design You are an expert in designing rigorous A/B experiments that produce actionable results. ## What You Do You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor. ## Test Structure ### 1. Hypothesis Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].' ### 2. Variants - Control (A): current design - Treatment (B): proposed change - Keep changes isolated — test one variable at a time ### 3. Primary Metric The single most important measure of success. Must be measurable, relevant, and sensitive to the change. ### 4. Secondary Metrics Supporting measures and guardrail metrics to detect unintended consequences. ### 5. Sample Size Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%). ### 6. Duration Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically. ## Common Pitfalls - Peeking at results before completion - Too many variants at once - Metric not sensitive enough to detect change - Sample size too small - Not accounting for novelty effects - Ignoring segmentation effects ## When Not to A/B Test - Very low traffic (insufficient sample) - Ethical concerns with withholding improvement - Foundational changes that affect everything - When qualitative insight is more valuable ## Best Practices - One hypothesis per test
Details
- Skill name
a-b-test-design- Repository
- sethdford/claude-skills
- Path
designer/prototyping-testing/skills/a-b-test-design/SKILL.md- License
- MIT
- Repo updated
- 11 March 2026
FAQ
What does the A B Test Design skill do?
Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.
How do I install the A B Test Design skill?
Claude Code plugin: /plugin marketplace add sethdford/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is A B Test Design popular?
It lives in sethdford/claude-skills, which has 48 GitHub stars and 10 forks.
Related skills
A B Test Design Pm
48sethdford/claude-skills
Analytics and experimentation skill for product managers.
Abuse Case Design
48sethdford/claude-skills
Design abuse cases (negative use cases) showing how attackers misuse system features. Use when identifying attacks that exploit intended functionality or business logic flaws.
Accessibility Audit
48sethdford/claude-skills
Conduct a comprehensive accessibility audit against WCAG guidelines with severity ratings and remediation steps.
Testing Python
49UCSB-NLP-Chang/Skill-Usage
Write and evaluate effective Python tests using pytest. Use when writing tests, reviewing test code, debugging test failures, or improving test coverage. Covers test design, fixtures, parameterization, mocking, and async testing.
Accessibility Test Plan
48sethdford/claude-skills
Create accessibility testing plans covering assistive technologies and WCAG criteria.
Excitation Signal Design
49UCSB-NLP-Chang/Skill-Usage
Design effective excitation signals (step tests) for system identification and parameter estimation in control systems.
Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.