Ab Test Analysis

Use when designing, checking, or interpreting A/B tests and experiments. Computes significance, confidence intervals, sample ratio mismatch, power, CUPED, and sample size with a bundled zero-dependency script, then judges validity risks, guardrails, and ship/iterate/rollback decisions and renders an HTML experiment report. 中文触发词:A/B 实验、AB 测试、实验解读、显著性、样本量计算、能不能全量。

9 starsTesting & QA

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add shisuidata/ss-data-skills --skill ab-test-analysis
Manual install
git clone --depth 1 https://github.com/shisuidata/ss-data-skills.git /tmp/ss-data-skills
mkdir -p ~/.claude/skills
cp -r /tmp/ss-data-skills/skills/ab-test-analysis ~/.claude/skills/ab-test-analysis

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# A/B Test Analysis

## 目标

设计、审查或解读 A/B 实验,判断实验是否可信、效果是否真实、业务上是否值得推广。

核心原则:**数字由脚本算,判断由人做**。显著性、置信区间、SRM、功效和样本量全部用 `scripts/ab_test.py` 计算,不心算、不估算;模型负责检查实验设计、解释结果、补充业务判断。

## 使用场景

- 实验前:确定样本量和实验周期(`plan`)
- 实验中:有人想“看一眼结果”,判断能不能提前停
- 实验后:解读结果,判断全量、继续观察、回滚还是重做
- 审查别人写的实验结论是否站得住
- 排查分流异常、结果离谱、指标打架

典型输入:

```text
这个实验转化率提升了 0.3 个百分点,p 值 0.04,可以全量吗?
```

## 不适用场景

- 没有随机分组的前后对比、自然实验:请使用 `business-root-cause-analysis`,并明确说明这不是实验
- 普通漏斗分析:使用 `funnel-analysis`
- 留存分析:使用 `retention-cohort-analysis`
- 指标定义本身有争议:先用 `metric-definition-reviewer`

没有对照组、实验组和核心指标数据时,不要给实验结论。

## 输入信息

最少输入(汇总数据即可):

- 每组样本量
- 核心指标的结果:比例指标给成功数,均值指标给均值和标准差,比率指标(如点击率)给用户粒度的分子和分母
- 实验天数

推荐输入:

- 实验假设、分流比例、计划周期
- 护栏指标结果(错误率、性能、投诉、退款等)及可接受的恶化幅度
- 预期分流比例(非均分实验必须提供,否则无法判断 SRM)
- 用户粒度明细 CSV(可做 CUPED 方差缩减)
- 实验前同一指标的数据(CUPED 协变量)

输入字段详见 [输入格式](references/input-schema.md)。

## 上下文建议

推荐使用 [实验分析上下文模板](../../context/templates/experiment-context.md)。

最有价值的上下文是:分流方式、计划周期、护栏指标、实验期间是否有大促或版本发布。缺少护栏指标时,不能建议直接全量。

## 工作流程

1. **整理输入**:把用户给的数字写成 `summary.json`(格式见 references),或直接使用用户的 CSV。
2. **运行脚本**(能执行代码时必须运行):

   ```bash
   python3 scripts/ab_test.py analyze --summary summary.json --html ab_report.html
   # 或用户粒度数据:
   python3 scripts/ab_test.py analyze --csv users.csv --variant-col variant --control A \
     --metric paid:proportion --metric gmv:mean --guardrail refund:proportion:lower \
     --cuped gmv=pre_gmv --days 14 --html ab_report.html
   ```
Read the full skill on GitHub

Details

Skill name
ab-test-analysis
Path
skills/ab-test-analysis/SKILL.md
License
MIT
Repo updated
8 November 2026

FAQ

What does the Ab Test Analysis skill do?

Use when designing, checking, or interpreting A/B tests and experiments. Computes significance, confidence intervals, sample ratio mismatch, power, CUPED, and sample size with a bundled zero-dependency script, then judges validity risks, guardrails, and ship/iterate/rollback decisions and renders an HTML experiment report. 中文触发词:A/B 实验、AB 测试、实验解读、显著性、样本量计算、能不能全量。

How do I install the Ab Test Analysis skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add shisuidata/ss-data-skills --skill ab-test-analysis. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Ab Test Analysis popular?

It lives in shisuidata/ss-data-skills, which has 9 GitHub stars and 0 forks.

Related skills

Business Root Cause Analysis

9

shisuidata/ss-data-skills

Use when diagnosing business metric changes or anomalies such as GMV decline, conversion drop, retention loss, churn increase, revenue change, lead decline, or active user fluctuation with a structured root cause analysis plan. 中文触发词:指标归因、为什么下降、异常排查、GMV 下滑、指标波动原因。

Marketing, Sales & Business

Daily Report Writer

9

shisuidata/ss-data-skills

Use when writing a concise daily report, daily worklog, standup update, shift handoff, or end-of-day summary for data analysis, data development, BI, governance, or operations work. 中文触发词:日报、日终同步、站会发言、值班交接。

Writing & Content

Dashboard Reviewer

9

shisuidata/ss-data-skills

Use when reviewing whether a BI dashboard, management dashboard, metric board, or data product page can support business decisions with clear metrics, trustworthy definitions, and actionable drill-down paths. 中文触发词:看板审查、BI 看板优化、经营看板改版、报表没人看。

Data & Analytics

Local Dev Testing

9

zllovesuki/cloudflare-agent-skills

Develop and test Cloudflare Workers locally with wrangler dev, Miniflare-compatible bindings, fixtures, integration tests, remote validation, and CI checks. Use when building local development workflows for Workers, D1, Durable Objects, KV, R2, Queues, or Workflows.

Testing & QA

Testing

9

sukirman1901/frontend-expert

Test user-facing UI with TDD, component tests, a11y checks, and critical-path coverage. Use when writing or fixing UI tests, verifying loading/error/empty states, or when the user mentions test/TDD/coverage/regresi — including natural language, not only /test-ui.

Testing & QA

Paddle Sandbox Testing

9

PaddleHQ/paddle-agent-skills

Test a Paddle integration end-to-end using the sandbox environment, test cards, the webhook simulator, and local tunnels — without taking real money.

Testing & QA

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.