Ab Test Analysis

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

27,559 stars 2,851 forksTesting & QA

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add phuryn/pm-skills --skill ab-test-analysis
Manual install
git clone --depth 1 https://github.com/phuryn/pm-skills.git /tmp/pm-skills
mkdir -p ~/.claude/skills
cp -r /tmp/pm-skills/pm-data-analytics/skills/ab-test-analysis ~/.claude/skills/ab-test-analysis

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

## A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

### Context

You are analyzing A/B test results for **$ARGUMENTS**.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

### Instructions

1. **Understand the experiment**:
   - What was the hypothesis?
   - What was changed (the variant)?
   - What is the primary metric? Any guardrail metrics?
   - How long did the test run?
   - What is the traffic split?

2. **Validate the test setup**:
   - **Sample size**: Is the sample large enough for the expected effect size?
     - Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
     - Flag if the test is underpowered (<80% power)
   - **Duration**: Did the test run for at least 1-2 full business cycles?
   - **Randomization**: Any evidence of sample ratio mismatch (SRM)?
   - **Novelty/primacy effects**: Was there enough time to wash out initial behavior changes?

3. **Calculate statistical significance**:
   - **Conversion rate** for control and variant
   - **Relative lift**: (variant - control) / control × 100
   - **p-value**: Using a two-tailed z-test or chi-squared test
   - **Confidence interval**: 95% CI for the difference
   - **Statistical significance**: Is p < 0.05?
   - **Practical significance**: Is the lift meaningful for the business?
Read the full skill on GitHub

Details

Skill name
ab-test-analysis
Repository
phuryn/pm-skills
Path
pm-data-analytics/skills/ab-test-analysis/SKILL.md
License
MIT
Repo updated
29 October 2026
Listed in
heilcheng/awesome-agent-skills, skillmatic-ai/awesome-agent-skills, VoltAgent/awesome-agent-skills

FAQ

What does the Ab Test Analysis skill do?

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

How do I install the Ab Test Analysis skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add phuryn/pm-skills --skill ab-test-analysis. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Ab Test Analysis popular?

It lives in phuryn/pm-skills, which has 27,559 GitHub stars and 2,851 forks, and is included in 3 curated skill lists.

Related skills

Analyze Feature Requests

28k

phuryn/pm-skills

Analyze and prioritize a list of feature requests by theme, strategic alignment, impact, effort, and risk. Use when reviewing customer feature requests, triaging a backlog, or making prioritization decisions.

Frontend & DesignIn 3 lists2.9k

Ansoff Matrix

28k

phuryn/pm-skills

Generate an Ansoff Matrix analysis mapping growth strategies across market penetration, market development, product development, and diversification. Use when considering growth options, planning market expansion, or evaluating strategic growth paths.

Marketing, Sales & BusinessIn 3 lists2.9k

Beachhead Segment

28k

phuryn/pm-skills

Identify the first beachhead market segment for a product launch. Evaluates segments against burning pain, willingness to pay, winnable market share, and referral potential. Use when choosing a first market, targeting an initial customer segment, or planning market entry strategy.

Marketing, Sales & BusinessIn 3 lists2.9k

Brainstorm Experiments Existing

28k

phuryn/pm-skills

Design experiments to test assumptions for an existing product — prototypes, A/B tests, spikes, and other low-effort validation methods. Use when validating assumptions, testing feature ideas cheaply, or planning product experiments.

Testing & QAIn 3 lists2.9k

Playwright

28k

openai/skills

Use when the task requires automating a real browser from the terminal (navigation, form filling, snapshots, screenshots, data extraction, UI-flow debugging) via `playwright-cli` or the bundled wrapper script.

Testing & QAIn 2 lists2k

Brainstorm Experiments New

28k

phuryn/pm-skills

Design lean startup experiments (pretotypes) for a new product. Creates XYZ hypotheses and suggests low-effort validation methods like landing pages, explainer videos, and pre-orders. Use when validating a new product idea, creating pretotypes, or testing market demand.

Testing & QAIn 3 lists2.9k

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.