Stat Ab Testing

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

248 stars 30 forksTesting & QA

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add asgard-ai-platform/skills --skill stat-ab-testing
Manual install
git clone --depth 1 https://github.com/asgard-ai-platform/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/stat-ab-testing ~/.claude/skills/stat-ab-testing

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# A/B Testing Statistics

## Framework

```
IRON LAW: Calculate Sample Size BEFORE Running the Test

Running a test without knowing the required sample size leads to two
failures: stopping too early (false positives) or running too long (waste).

Required inputs: baseline conversion rate, minimum detectable effect (MDE),
significance level (α), power (1-β). Calculate BEFORE starting.
```

### Sample Size Formula (Proportions)

```
n per group ≈ (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²
```

Quick reference (α=0.05, power=0.8):
| Baseline Rate | MDE (relative) | N per Group |
|--------------|----------------|-------------|
| 5% | 10% (→5.5%) | ~58,000 |
| 5% | 20% (→6.0%) | ~15,000 |
| 10% | 10% (→11%) | ~15,000 |
| 10% | 20% (→12%) | ~4,000 |

### Testing Approaches

| Approach | How It Works | Best When |
|----------|-------------|-----------|
| **Frequentist (fixed-horizon)** | Set sample size, run to completion, then analyze | Standard practice, well-understood |
| **Bayesian** | Update beliefs with data, compute probability of improvement | Want probability statements ("90% chance B is better") |
| **Sequential testing** | Check results at intervals with adjusted thresholds | Need to stop early if clear winner, or limit downside risk |

### Experiment Design Checklist

1. **Hypothesis**: What do you expect to happen and why?
2. **Primary metric**: ONE key metric (conversion, revenue, retention)
Read the full skill on GitHub

Details

Skill name
stat-ab-testing
Path
stat-ab-testing/SKILL.md
License
MIT
Repo updated
6 June 2026

FAQ

What does the Stat Ab Testing skill do?

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

How do I install the Stat Ab Testing skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add asgard-ai-platform/skills --skill stat-ab-testing. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Stat Ab Testing popular?

It lives in asgard-ai-platform/skills, which has 248 GitHub stars and 30 forks.

Related skills

Algo Ad Bidding

248

asgard-ai-platform/skills

Implement and select ad bidding strategies from manual CPC to automated target-CPA and target-ROAS. Use this skill when the user needs to choose a bidding strategy, set up automated bidding, or optimize bid parameters — even if they say 'what bidding strategy should I use', 'target CPA setup', or 'smart bidding configuration'.

Marketing, Sales & Business30

Algo Ad Budget

248

asgard-ai-platform/skills

Optimize advertising budget allocation across campaigns using marginal returns analysis. Use this skill when the user needs to distribute budget across multiple campaigns, optimize spend pacing, or maximize overall ROAS under budget constraints — even if they say 'how to split my ad budget', 'campaign budget optimization', or 'diminishing returns on ad spend'.

Marketing, Sales & Business30

Algo Ad Ctr

248

asgard-ai-platform/skills

Build CTR prediction models for estimating ad click-through rates from features. Use this skill when the user needs to predict click probability, build an ad ranking model, or evaluate ad creative performance — even if they say 'predict click rate', 'ad relevance scoring', or 'which ad will get more clicks'.

Marketing, Sales & Business30

Stat Hypothesis Testing

248

asgard-ai-platform/skills

Conduct statistical hypothesis testing including null/alternative hypothesis formulation, p-values, Type I/II errors, and test statistic selection. Use this skill when the user needs to determine whether a result is statistically significant, choose the right statistical test, interpret p-values correctly, or evaluate research findings — even if they say 'is this result significant', 'which statistical test should I use', or 'what does this p-value mean'.

Testing & QA30

Msp Maintenance

121

RTFM-IT-Services-LLC/msp-claude-skills

Use this skill for your MSP's proactive, recurring operations: patching and update cycles, maintenance windows, backup monitoring and test restores, monitoring and alert triage, the on-call rotation, change management for risky changes, documentation currency, and vendor or license renewals. Trigger on "patch window", "maintenance window", "did the backups run", "test restore", "alert storm", "who is on call", "can we make this change", "update the firewall", or any question about the recurring delivery work between tickets. msp-helpdesk owns the reactive desk (tickets, priorities, response ta

Testing & QA37

Browser Uat Analyst

110

microsoft/cat-agent-skills

Use this skill whenever the user asks to test, validate, UAT, regression-check, or investigate the behaviour of a browser-based application — including Copilot Studio agents, Power Platform apps and flows, Dataverse, Dynamics 365, Microsoft 365 admin centres, Teams web, and custom web apps. Covers test charter design, Playwright-driven execution, screenshot evidence capture, expected-vs-observed analysis, and stakeholder reporting. Use it BEFORE claiming any UI behaviour works or is broken. Do NOT use this skill for unit tests, API-only contract tests, or load testing.

Testing & QA122

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.