Data Science Experimentation

Use when the user asks whether an A/B test result is statistically significant, sample size and power for an experiment, CUPED, sequential testing, sample ratio mismatch, multiple comparisons or bandits. Produces design, power calculation and analysis. Not for product A/B test strategy (use product-ab-testing) or observational causal studies (use data-science-causal-inference).

45 stars 6 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill data-science-experimentation
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/data-science/experimentation ~/.claude/skills/data-science-experimentation

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Experimentation

## Purpose
Design and analyze rigorous A/B tests and experiments. Enforce proper sample size planning, pre-registration, statistical methods, guardrail monitoring, and decision frameworks.

## Agent Protocol

### Trigger
Exact user phrases: "A/B test", "experiment", "randomized trial", "sample size", "power analysis", "hypothesis test", "p-value", "multiple testing", "guardrail metric", "experiment design", "treatment effect", "H0", "alternative hypothesis", "statistical significance", "practical significance", "MDE", "minimum detectable effect".

### Input Context
- Experiment type (A/B, multivariate, MAB, switchback)
- Primary metric(s) and their baseline values
- Minimum detectable effect (MDE) in absolute or relative terms
- Traffic volume and expected experiment duration
- Unit of randomization (user, session, cluster)
- Guardrail metrics and acceptable thresholds
- Regulatory and ethical considerations
- Existing experiment platform and tooling

### Output Artifact
Experiment design with sample size calculation, analysis plan, and decision criteria.

### Response Format
```
## Experiment Design
Hypothesis: {H0 and H1}
Primary Metric: {name, baseline, MDE}
Sample Size: {N per variant, total N}
Duration: {days} at {traffic allocation}
Analysis: {test type, corrections, covariates}

## Analysis Results
Treatment Effect: {estimate, CI, p-value}
Practical Significance: {effect size, decision}
Guardrails: {√ or ✗ per metric}
```

### Completion Criteria
Read the full skill on GitHub

Details

Skill name
data-science-experimentation
Path
skills/data-science/experimentation/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Data Science Experimentation skill do?

Use when the user asks whether an A/B test result is statistically significant, sample size and power for an experiment, CUPED, sequential testing, sample ratio mismatch, multiple comparisons or bandits. Produces design, power calculation and analysis. Not for product A/B test strategy (use product-ab-testing) or observational causal studies (use data-science-causal-inference).

How do I install the Data Science Experimentation skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill data-science-experimentation. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Science Experimentation popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Data Science Statistical Analysis

45

j4flmao/agent-skills

Use when the user asks about hypothesis tests (t-test, chi-square, Mann-Whitney), regression diagnostics, effect sizes, confidence intervals, Bayesian inference with PyMC, missing data or exploratory statistics. Produces the right test or model, assumption checks and reported effect sizes. Not for experiment design (use data-science-experimentation) or forecasting (use ml-time-series).

Data & Analytics

Data Science Analytics Engineering

45

j4flmao/agent-skills

Use when the user asks about analytics engineering with dbt: staging/intermediate/mart models, materializations, incremental models, Jinja macros, the dbt Semantic Layer / MetricFlow, Cube or analytical SQL. Produces dbt project layout, models, tests and metrics. Not for Kimball grain and SCD design (use data-dimensional-modeling) or Airflow wiring (use data-airflow-dbt-orchestration).

Data & Analytics

Ecmwf Earthkit

23

ecmwf/ecmwf-weather-skills

Processes, computes and plots ECMWF weather and climate data with ECMWF's earthkit components — load it before writing ANY numpy, xarray, pandas, matplotlib or cartopy code for weather data, and for any local GRIB (.grib, .grib2) or NetCDF file. Covers reading and selecting fields (earthkit-data), ECMWF-styled maps, multi-panel figures, meteograms and plumes (earthkit-plots), ensemble mean/spread/percentiles, daily and monthly statistics, de-accumulation, climatologies, anomalies and area or country means (earthkit-transforms), wind, humidity, dewpoint, potential temperature, wet bulb, EFI and

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.