Monitor Model Drift
Implement model drift monitoring using Evidently AI, statistical tests (PSI, KS), and custom metrics to detect data drift and concept drift in production ML systems. Set up automated alerting and reporting workflows to catch degradation before it impacts business metrics. Use when production models show unexplained performance degradation, when new data distributions differ from training data, when seasonal shifts affect input features, or when regulatory requirements mandate model monitoring.
Install
npx skills add pjt222/agent-almanac --skill monitor-model-driftgit clone --depth 1 https://github.com/pjt222/agent-almanac.git /tmp/agent-almanac
mkdir -p ~/.claude/skills
cp -r /tmp/agent-almanac/i18n/caveman-lite/skills/monitor-model-drift ~/.claude/skills/monitor-model-driftPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Monitor Model Drift
> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.
Detect and alert on data drift and concept drift in production ML models using statistical tests and automated monitoring.
## When to Use
- Production ML models with unexplained performance degradation
- New data distributions differ from training data
- Seasonal or temporal shifts in input features
- Need proactive alerts before business metrics are impacted
- Regulatory requirements for model monitoring (e.g., SR 11-7, EU AI Act)
- Multiple model versions deployed requiring drift comparison
## Inputs
- **Required**: Production model predictions and features (last 30-90 days)
- **Required**: Reference dataset (training or validation data)
- **Required**: Ground truth labels (may be delayed)
- **Optional**: Feature importance scores or SHAP values
- **Optional**: Business metric thresholds for alerting
- **Optional**: Historical drift reports for trend analysis
## Procedure
### Step 1: Install and Configure Evidently AI
Set up the monitoring framework with appropriate dependencies.
```bash
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install Evidently and dependencies
pip install evidently pandas scikit-learn prometheus-client
# Create monitoring directory structure
mkdir -p monitoring/{reports,config,alerts}
```
Create configuration file:Details
- Skill name
monitor-model-drift- Repository
- pjt222/agent-almanac
- Path
i18n/caveman-lite/skills/monitor-model-drift/SKILL.md- License
- MIT
- Repo updated
- 18 November 2026
- Listed in
- BehiSecc/awesome-claude-skills
FAQ
What does the Monitor Model Drift skill do?
Implement model drift monitoring using Evidently AI, statistical tests (PSI, KS), and custom metrics to detect data drift and concept drift in production ML systems. Set up automated alerting and reporting workflows to catch degradation before it impacts business metrics. Use when production models show unexplained performance degradation, when new data distributions differ from training data, when seasonal shifts affect input features, or when regulatory requirements mandate model monitoring.
How do I install the Monitor Model Drift skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add pjt222/agent-almanac --skill monitor-model-drift. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Monitor Model Drift popular?
It lives in pjt222/agent-almanac, which has 45 GitHub stars and 4 forks, and is included in 1 curated skill list.
Related skills
Adapt Architecture
45pjt222/agent-almanac
Execute structural metamorphosis using strangler fig migration, chrysalis phases, and interface preservation. Covers transformation planning, parallel running, progressive cutover, rollback design, and post-metamorphosis stabilization for system architecture evolution. Use when assess-form has classified the system as READY for transformation, when migrating from monolith to microservices, when replacing a core subsystem while dependents continue operating, or when any architectural change must be gradual rather than big-bang.
Adaptic
45pjt222/agent-almanac
Master skill composing the 5-step synoptic cycle for panoramic synthesis across multiple domains. Orchestrates meditate, expand-awareness, observe, awareness, integrate-gestalt, and express-insight into a coherent process that produces unified understanding rather than sequential compromise. Use when a problem genuinely spans 3+ domains and the interactions between domains matter more than depth in any one, when sequential analysis feels like compromise rather than integration, or before major architectural decisions affecting multiple stakeholders.
Add Puzzle Type
45pjt222/agent-almanac
Scaffold a new puzzle type across all 10+ pipeline integration points in jigsawR. Creates the core puzzle module, wires it into the unified pipeline (generation, positioning, rendering, adjacency), adds ggpuzzle geom/stat layers, updates DESCRIPTION and config.yml, extends the Shiny app, and creates a comprehensive test suite. Use when adding a completely new puzzle type to the package or following the 10-point integration checklist to ensure nothing is missed end-to-end.
Orchestrate Ml Pipeline
45pjt222/agent-almanac
Orchestrate end-to-end machine learning pipelines using Prefect or Airflow with DAG construction, task dependencies, retry logic, scheduling, monitoring, and integration with MLflow, DVC, and feature stores for production ML workflows. Use when automating multi-step ML workflows from data ingestion to deployment, scheduling periodic model retraining, coordinating distributed training tasks, or managing retry logic and failure recovery across pipeline stages.
Model Markov Chain
45pjt222/agent-almanac
Build and analyze discrete or continuous Markov chains including transition matrix construction, state classification, stationary distribution computation, and mean first passage times. Use when modeling a memoryless system with observed transition counts or rates, computing long-run steady-state probabilities, determining expected hitting times or absorption probabilities, classifying states as transient or recurrent, or building a foundation for hidden Markov models or reinforcement learning MDPs.
Policy Starter
43AlsKozlov/ru-legal
Draft AI Policy для организации. РФ specifics: integrate с 152-ФЗ положение о ПДн; sectoral overlays; пилотные regulatory sandboxes (ФЗ-258); national AI strategy (Указ 490/2019) принципы; sanctions context. Template для Acceptable Use + Risk Management + Governance bodies.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.