A B Test Design

Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.

3,414 stars 447 forksTesting & QA Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add Owl-Listener/designer-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add Owl-Listener/designer-skills --skill a-b-test-design
Manual install
git clone --depth 1 https://github.com/Owl-Listener/designer-skills.git /tmp/designer-skills
mkdir -p ~/.claude/skills
cp -r /tmp/designer-skills/prototyping-testing/skills/a-b-test-design ~/.claude/skills/a-b-test-design

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
## What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
## Test Structure
### 1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
### 2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
### 3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
### 4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
### 5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
### 6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
## Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
## When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
## Best Practices
- One hypothesis per test
Read the full skill on GitHub

Details

Skill name
a-b-test-design
Path
prototyping-testing/skills/a-b-test-design/SKILL.md
License
MIT
Repo updated
5 September 2026

FAQ

What does the A B Test Design skill do?

Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.

How do I install the A B Test Design skill?

Claude Code plugin: /plugin marketplace add Owl-Listener/designer-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is A B Test Design popular?

It lives in Owl-Listener/designer-skills, which has 3,414 GitHub stars and 447 forks.

Related skills

Accessibility Audit

3.4k

Owl-Listener/designer-skills

Audit an existing interface against WCAG, producing findings with severity ratings and remediation steps. Use when you have a design or build to assess now. Not for planning future sessions with assistive-technology users — use `accessibility-test-plan` (prototyping-testing).

Frontend & Design447

Accessibility Test Plan

3.4k

Owl-Listener/designer-skills

Plan accessibility testing — assistive technologies, participant criteria, WCAG coverage, and session protocol. Use when scheduling testing with real AT users. Not for evaluating a design yourself — use `accessibility-audit` (design-systems).

Testing & QA447

Aesthetic Usability

3.4k

Owl-Listener/designer-skills

Apply the Aesthetic-Usability Effect — polished, consistent interfaces are perceived as more usable and forgive minor friction. Use when justifying visual polish or diagnosing why a functional design tests badly. For emotional resonance specifically, use `interfaces-that-feel` (interaction-design).

Testing & QA447

Stats Integrity

2.1k

ai4s-research/open-science

Use whenever you run statistical analysis for the social sciences (regression, hypothesis tests, econometrics) or read Stata (.dta) / SPSS (.sav) data in this workspace. Enforces an execute-don't-interpret boundary (surface estimates, don't volunteer causal claims), checks the analysis against a preregistration plan for HARKing, verifies reproducible seeds, and reproduces .dta/.sav estimates via R. Flags integrity risks; never certifies the analysis is sound.

Testing & QA231

Steve Jobs Design Review

2.7k

wondelai/skills

Review designs, products, and features with Steve Jobs' standards: ruthless simplicity, focus, and end-to-end excellence. Use when the user mentions "Steve Jobs review", "design review", "product review", "what would Steve do", "insanely great", "this feels too complicated", "too many features", "product taste", "saying no", or "is this good enough to ship". Also trigger when critiquing a UI, feature, or roadmap for focus and simplicity, cutting scope to the essential, or pressure-testing the whole experience from first run to daily use. Covers the simplicity audit, the no list, design-is-how-

Testing & QAIn 2 lists266

Click Test Plan

3.4k

Owl-Listener/designer-skills

Design first-click and click tests for findability and navigation. Use when testing whether people can locate something. For full task-based observation, use `test-scenario`.

Testing & QA447

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.