Ab Test Analysis

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

9 stars 2 forksTesting & QA Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add tarunccet/pm-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add tarunccet/pm-skills --skill ab-test-analysis
Manual install
git clone --depth 1 https://github.com/tarunccet/pm-skills.git /tmp/pm-skills
mkdir -p ~/.claude/skills
cp -r /tmp/pm-skills/pm-data-analytics/skills/ab-test-analysis ~/.claude/skills/ab-test-analysis

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

## A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

### Context

You are analyzing A/B test results for **$ARGUMENTS**.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

### Instructions

### Phase 0: Context Confirmation

Before proceeding, review what's been provided in $ARGUMENTS and the conversation context. **Only ask about what's missing — don't re-ask what's already clear.**

1. **Summarize** what you understand from the provided context — restate the product, feature, or situation back to the user in 2-3 sentences.
2. **Identify gaps** — if any of the following are unclear, ask:
   - What was the hypothesis being tested?
   - What is the primary metric and any guardrail metrics?
   - How long did the test run and what was the traffic split?
3. **Confirm**: _"Here's my understanding: [summary]. I plan to [brief description of what the skill will produce]. Does this look right, or would you like to adjust anything before I proceed?"_

If the user provides additional context, incorporate it. If the user confirms, proceed.

1. **Understand the experiment**:
   - What was the hypothesis?
   - What was changed (the variant)?
   - What is the primary metric? Any guardrail metrics?
   - How long did the test run?
   - What is the traffic split?

2. **Validate the test setup**:
Read the full skill on GitHub

Details

Skill name
ab-test-analysis
Path
pm-data-analytics/skills/ab-test-analysis/SKILL.md
License
MIT
Repo updated
13 November 2026

FAQ

What does the Ab Test Analysis skill do?

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

How do I install the Ab Test Analysis skill?

Claude Code plugin: /plugin marketplace add tarunccet/pm-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Ab Test Analysis popular?

It lives in tarunccet/pm-skills, which has 9 GitHub stars and 2 forks.

Related skills

Adr

9

tarunccet/pm-skills

Document architecture and product decisions using the Nygard ADR format — captures context, alternatives considered, and consequences so the team never asks 'why did we do it this way?' again. Use when making architecture decisions, choosing between technical approaches, selecting vendors, or any decision that will be hard to reverse.

Software Development

AI Build Buy Partner

9

tarunccet/pm-skills

Evaluate AI capability sourcing options across build, buy, fine-tune, and partner archetypes using a structured decision matrix. Use when deciding whether to build a custom model, use an LLM API, fine-tune a base model, or partner with an AI vendor.

AI, LLMs & Agents

AI Data Strategy

9

tarunccet/pm-skills

Define a data strategy for AI products covering training data requirements, data quality, labeling strategy, feedback loops, and continuous retraining. Use when assessing data readiness for an AI feature, designing a data flywheel, or planning an ML data pipeline.

AI, LLMs & Agents

Brainstorm Experiments Existing

9

tarunccet/pm-skills

Design experiments to test assumptions for an existing product — prototypes, A/B tests, spikes, and other low-effort validation methods. Use when validating assumptions, testing feature ideas cheaply, or planning product experiments.

Testing & QA

Test Targeted

9

yasunori0418/skills

テスト実行時の絞り込み運用ルール。プロジェクト全体テストではなく修正範囲に絞ったテスト実行を行う。`gradlew test`/`pytest`/`jest`/`go test`等のテストランナーを実行するとき、テスト実行コマンドを組み立てるときに必ず参照する。コミット差分や修正ファイルから対象テストを特定する手順を含む。

Testing & QA

Brainstorm Experiments New

9

tarunccet/pm-skills

Design lean startup experiments (pretotypes) for a new product. Creates XYZ hypotheses and suggests low-effort validation methods like landing pages, explainer videos, and pre-orders. Use when validating a new product idea, creating pretotypes, or testing market demand.

Testing & QA

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.