Run Ab Test Models
Design and execute A/B tests for ML models in production using traffic splitting, statistical significance testing, and canary/shadow deployment. Measure performance differences and make data-driven rollout decisions. Use to validate a new model before full rollout, compare candidate models from different algorithms, measure business metric impact of model changes, or meet regulatory gradual rollout requirements.
Install
npx skills add pjt222/agent-almanac --skill run-ab-test-modelsgit clone --depth 1 https://github.com/pjt222/agent-almanac.git /tmp/agent-almanac
mkdir -p ~/.claude/skills
cp -r /tmp/agent-almanac/i18n/caveman-lite/skills/run-ab-test-models ~/.claude/skills/run-ab-test-modelsPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Run A/B Test for Models > See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates. Execute controlled experiments comparing model versions using traffic splitting and statistical analysis. ## When to Use - Deploying a new model version and validating improvement before full rollout - Comparing multiple candidate models from different algorithms or features - Testing impact of hyperparameter changes on business metrics - Measuring model performance in production without risking full traffic - Regulatory requirements for gradual rollout (e.g., medical ML) - Evaluating cost-performance tradeoffs between model sizes ## Inputs - **Required**: Champion model (current production version) - **Required**: Challenger model(s) (new version to test) - **Required**: Traffic allocation percentage (e.g., 5% to challenger) - **Required**: Success metrics (business and ML) - **Required**: Minimum sample size or test duration - **Optional**: Guardrail metrics (latency, error rate thresholds) - **Optional**: User segments for stratified testing ## Procedure ### Step 1: Design Experiment Define test parameters, success criteria, and statistical requirements. ```python # ab_test/experiment_config.py from dataclasses import dataclass from typing import List, Dict import numpy as np from scipy.stats import norm @dataclass # ... (see EXAMPLES.md for complete implementation) ```
Details
- Skill name
run-ab-test-models- Repository
- pjt222/agent-almanac
- Path
i18n/caveman-lite/skills/run-ab-test-models/SKILL.md- License
- MIT
- Repo updated
- 18 November 2026
- Listed in
- BehiSecc/awesome-claude-skills
FAQ
What does the Run Ab Test Models skill do?
Design and execute A/B tests for ML models in production using traffic splitting, statistical significance testing, and canary/shadow deployment. Measure performance differences and make data-driven rollout decisions. Use to validate a new model before full rollout, compare candidate models from different algorithms, measure business metric impact of model changes, or meet regulatory gradual rollout requirements.
How do I install the Run Ab Test Models skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add pjt222/agent-almanac --skill run-ab-test-models. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Run Ab Test Models popular?
It lives in pjt222/agent-almanac, which has 45 GitHub stars and 4 forks, and is included in 1 curated skill list.
Related skills
Adapt Architecture
45pjt222/agent-almanac
Execute structural metamorphosis using strangler fig migration, chrysalis phases, and interface preservation. Covers transformation planning, parallel running, progressive cutover, rollback design, and post-metamorphosis stabilization for system architecture evolution. Use when assess-form has classified the system as READY for transformation, when migrating from monolith to microservices, when replacing a core subsystem while dependents continue operating, or when any architectural change must be gradual rather than big-bang.
Adaptic
45pjt222/agent-almanac
Master skill composing the 5-step synoptic cycle for panoramic synthesis across multiple domains. Orchestrates meditate, expand-awareness, observe, awareness, integrate-gestalt, and express-insight into a coherent process that produces unified understanding rather than sequential compromise. Use when a problem genuinely spans 3+ domains and the interactions between domains matter more than depth in any one, when sequential analysis feels like compromise rather than integration, or before major architectural decisions affecting multiple stakeholders.
Add Puzzle Type
45pjt222/agent-almanac
Scaffold a new puzzle type across all 10+ pipeline integration points in jigsawR. Creates the core puzzle module, wires it into the unified pipeline (generation, positioning, rendering, adjacency), adds ggpuzzle geom/stat layers, updates DESCRIPTION and config.yml, extends the Shiny app, and creates a comprehensive test suite. Use when adding a completely new puzzle type to the package or following the 10-point integration checklist to ensure nothing is missed end-to-end.
Scaffold MCP Server
45pjt222/agent-almanac
Scaffold a new MCP server from tool specifications using the official SDK (TypeScript or Python), including transport configuration, tool handlers, and test harness. Use when you have a tool spec and need a working server, starting a new MCP project with correct structure, migrating an existing tool integration to the MCP protocol, or prototyping a tool surface to test with Claude Code before full implementation.
Register Ml Model
45pjt222/agent-almanac
Register trained models in MLflow Model Registry with version control, implement stage transitions (Staging, Production, Archived) with approval workflows, and manage model lineage with comprehensive metadata and deployment tracking. Use when promoting a trained model from experimentation to production, managing multiple model versions across development stages, implementing approval workflows for governance, rolling back to previous versions, or auditing model changes for compliance.
Setup Automl Pipeline
45pjt222/agent-almanac
Configure automated ML pipelines using Optuna or Ray Tune for hyperparameter optimization. Implement efficient search strategies (Hyperband, ASHA), define search spaces, and set up early stopping to find optimal model configurations with minimal manual tuning. Use when starting a new ML project, retraining with new data and re-optimizing hyperparameters, comparing multiple algorithms, or when the team lacks deep expertise in specific algorithm hyperparameters.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.