Monitor Model Drift

Implement model drift monitoring using Evidently AI, statistical tests (PSI, KS), and custom metrics to detect data drift and concept drift in production ML systems. Set up automated alerting and reporting workflows to catch degradation before it impacts business metrics. Use when production models show unexplained performance degradation, when new data distributions differ from training data, when seasonal shifts affect input features, or when regulatory requirements mandate model monitoring.

45 stars 4 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add pjt222/agent-almanac --skill monitor-model-drift
Manual install
git clone --depth 1 https://github.com/pjt222/agent-almanac.git /tmp/agent-almanac
mkdir -p ~/.claude/skills
cp -r /tmp/agent-almanac/i18n/caveman-lite/skills/monitor-model-drift ~/.claude/skills/monitor-model-drift

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Monitor Model Drift

> See [Extended Examples](references/EXAMPLES.md) for complete configuration files and templates.

Detect and alert on data drift and concept drift in production ML models using statistical tests and automated monitoring.

## When to Use

- Production ML models with unexplained performance degradation
- New data distributions differ from training data
- Seasonal or temporal shifts in input features
- Need proactive alerts before business metrics are impacted
- Regulatory requirements for model monitoring (e.g., SR 11-7, EU AI Act)
- Multiple model versions deployed requiring drift comparison

## Inputs

- **Required**: Production model predictions and features (last 30-90 days)
- **Required**: Reference dataset (training or validation data)
- **Required**: Ground truth labels (may be delayed)
- **Optional**: Feature importance scores or SHAP values
- **Optional**: Business metric thresholds for alerting
- **Optional**: Historical drift reports for trend analysis

## Procedure

### Step 1: Install and Configure Evidently AI

Set up the monitoring framework with appropriate dependencies.

```bash
# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install Evidently and dependencies
pip install evidently pandas scikit-learn prometheus-client

# Create monitoring directory structure
mkdir -p monitoring/{reports,config,alerts}
```

Create configuration file:
Read the full skill on GitHub

Details

Skill name
monitor-model-drift
Path
i18n/caveman-lite/skills/monitor-model-drift/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the Monitor Model Drift skill do?

Implement model drift monitoring using Evidently AI, statistical tests (PSI, KS), and custom metrics to detect data drift and concept drift in production ML systems. Set up automated alerting and reporting workflows to catch degradation before it impacts business metrics. Use when production models show unexplained performance degradation, when new data distributions differ from training data, when seasonal shifts affect input features, or when regulatory requirements mandate model monitoring.

How do I install the Monitor Model Drift skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add pjt222/agent-almanac --skill monitor-model-drift. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Monitor Model Drift popular?

It lives in pjt222/agent-almanac, which has 45 GitHub stars and 4 forks, and is included in 1 curated skill list.

Related skills

Adapt Architecture

45

pjt222/agent-almanac

Execute structural metamorphosis using strangler fig migration, chrysalis phases, and interface preservation. Covers transformation planning, parallel running, progressive cutover, rollback design, and post-metamorphosis stabilization for system architecture evolution. Use when assess-form has classified the system as READY for transformation, when migrating from monolith to microservices, when replacing a core subsystem while dependents continue operating, or when any architectural change must be gradual rather than big-bang.

Software DevelopmentIn 1 list

Adaptic

45

pjt222/agent-almanac

Master skill composing the 5-step synoptic cycle for panoramic synthesis across multiple domains. Orchestrates meditate, expand-awareness, observe, awareness, integrate-gestalt, and express-insight into a coherent process that produces unified understanding rather than sequential compromise. Use when a problem genuinely spans 3+ domains and the interactions between domains matter more than depth in any one, when sequential analysis feels like compromise rather than integration, or before major architectural decisions affecting multiple stakeholders.

OtherIn 1 list

Add Puzzle Type

45

pjt222/agent-almanac

Scaffold a new puzzle type across all 10+ pipeline integration points in jigsawR. Creates the core puzzle module, wires it into the unified pipeline (generation, positioning, rendering, adjacency), adds ggpuzzle geom/stat layers, updates DESCRIPTION and config.yml, extends the Shiny app, and creates a comprehensive test suite. Use when adding a completely new puzzle type to the package or following the 10-point integration checklist to ensure nothing is missed end-to-end.

Testing & QAIn 1 list

Orchestrate Ml Pipeline

45

pjt222/agent-almanac

Orchestrate end-to-end machine learning pipelines using Prefect or Airflow with DAG construction, task dependencies, retry logic, scheduling, monitoring, and integration with MLflow, DVC, and feature stores for production ML workflows. Use when automating multi-step ML workflows from data ingestion to deployment, scheduling periodic model retraining, coordinating distributed training tasks, or managing retry logic and failure recovery across pipeline stages.

AI, LLMs & AgentsIn 1 list

Model Markov Chain

45

pjt222/agent-almanac

Build and analyze discrete or continuous Markov chains including transition matrix construction, state classification, stationary distribution computation, and mean first passage times. Use when modeling a memoryless system with observed transition counts or rates, computing long-run steady-state probabilities, determining expected hitting times or absorption probabilities, classifying states as transient or recurrent, or building a foundation for hidden Markov models or reinforcement learning MDPs.

AI, LLMs & AgentsIn 1 list

Policy Starter

43

AlsKozlov/ru-legal

Draft AI Policy для организации. РФ specifics: integrate с 152-ФЗ положение о ПДн; sectoral overlays; пилотные regulatory sandboxes (ФЗ-258); national AI strategy (Указ 490/2019) принципы; sanctions context. Template для Acceptable Use + Risk Management + Governance bodies.

AI, LLMs & Agents

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.