Run Ab Test Models

Design and execute A/B tests for ML models in production using traffic splitting, statistical significance testing, and canary/shadow deployment. Measure performance differences and make data-driven rollout decisions. Use to validate a new model before full rollout, compare candidate models from different algorithms, measure business metric impact of model changes, or meet regulatory gradual rollout requirements.

45 stars 4 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add pjt222/agent-almanac --skill run-ab-test-models
Manual install
git clone --depth 1 https://github.com/pjt222/agent-almanac.git /tmp/agent-almanac
mkdir -p ~/.claude/skills
cp -r /tmp/agent-almanac/i18n/caveman-lite/skills/run-ab-test-models ~/.claude/skills/run-ab-test-models

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Run A/B Test for Models

> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.

Execute controlled experiments comparing model versions using traffic splitting and statistical analysis.

## When to Use

- Deploying a new model version and validating improvement before full rollout
- Comparing multiple candidate models from different algorithms or features
- Testing impact of hyperparameter changes on business metrics
- Measuring model performance in production without risking full traffic
- Regulatory requirements for gradual rollout (e.g., medical ML)
- Evaluating cost-performance tradeoffs between model sizes

## Inputs

- **Required**: Champion model (current production version)
- **Required**: Challenger model(s) (new version to test)
- **Required**: Traffic allocation percentage (e.g., 5% to challenger)
- **Required**: Success metrics (business and ML)
- **Required**: Minimum sample size or test duration
- **Optional**: Guardrail metrics (latency, error rate thresholds)
- **Optional**: User segments for stratified testing

## Procedure

### Step 1: Design Experiment

Define test parameters, success criteria, and statistical requirements.

```python
# ab_test/experiment_config.py
from dataclasses import dataclass
from typing import List, Dict
import numpy as np
from scipy.stats import norm

@dataclass
# ... (see EXAMPLES.md for complete implementation)
```
Read the full skill on GitHub

Details

Skill name
run-ab-test-models
Path
i18n/caveman-lite/skills/run-ab-test-models/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the Run Ab Test Models skill do?

Design and execute A/B tests for ML models in production using traffic splitting, statistical significance testing, and canary/shadow deployment. Measure performance differences and make data-driven rollout decisions. Use to validate a new model before full rollout, compare candidate models from different algorithms, measure business metric impact of model changes, or meet regulatory gradual rollout requirements.

How do I install the Run Ab Test Models skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add pjt222/agent-almanac --skill run-ab-test-models. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Run Ab Test Models popular?

It lives in pjt222/agent-almanac, which has 45 GitHub stars and 4 forks, and is included in 1 curated skill list.

Related skills

Adapt Architecture

45

pjt222/agent-almanac

Execute structural metamorphosis using strangler fig migration, chrysalis phases, and interface preservation. Covers transformation planning, parallel running, progressive cutover, rollback design, and post-metamorphosis stabilization for system architecture evolution. Use when assess-form has classified the system as READY for transformation, when migrating from monolith to microservices, when replacing a core subsystem while dependents continue operating, or when any architectural change must be gradual rather than big-bang.

Software DevelopmentIn 1 list

Adaptic

45

pjt222/agent-almanac

Master skill composing the 5-step synoptic cycle for panoramic synthesis across multiple domains. Orchestrates meditate, expand-awareness, observe, awareness, integrate-gestalt, and express-insight into a coherent process that produces unified understanding rather than sequential compromise. Use when a problem genuinely spans 3+ domains and the interactions between domains matter more than depth in any one, when sequential analysis feels like compromise rather than integration, or before major architectural decisions affecting multiple stakeholders.

OtherIn 1 list

Add Puzzle Type

45

pjt222/agent-almanac

Scaffold a new puzzle type across all 10+ pipeline integration points in jigsawR. Creates the core puzzle module, wires it into the unified pipeline (generation, positioning, rendering, adjacency), adds ggpuzzle geom/stat layers, updates DESCRIPTION and config.yml, extends the Shiny app, and creates a comprehensive test suite. Use when adding a completely new puzzle type to the package or following the 10-point integration checklist to ensure nothing is missed end-to-end.

Testing & QAIn 1 list

Scaffold MCP Server

45

pjt222/agent-almanac

Scaffold a new MCP server from tool specifications using the official SDK (TypeScript or Python), including transport configuration, tool handlers, and test harness. Use when you have a tool spec and need a working server, starting a new MCP project with correct structure, migrating an existing tool integration to the MCP protocol, or prototyping a tool surface to test with Claude Code before full implementation.

AI, LLMs & AgentsIn 1 list

Register Ml Model

45

pjt222/agent-almanac

Register trained models in MLflow Model Registry with version control, implement stage transitions (Staging, Production, Archived) with approval workflows, and manage model lineage with comprehensive metadata and deployment tracking. Use when promoting a trained model from experimentation to production, managing multiple model versions across development stages, implementing approval workflows for governance, rolling back to previous versions, or auditing model changes for compliance.

AI, LLMs & AgentsIn 1 list

Setup Automl Pipeline

45

pjt222/agent-almanac

Configure automated ML pipelines using Optuna or Ray Tune for hyperparameter optimization. Implement efficient search strategies (Hyperband, ASHA), define search spaces, and set up early stopping to find optimal model configurations with minimal manual tuning. Use when starting a new ML project, retraining with new data and re-optimizing hyperparameters, comparing multiple algorithms, or when the team lacks deep expertise in specific algorithm hyperparameters.

AI, LLMs & AgentsIn 1 list

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.