Config Drift Checker

config-drift-checker

CI for your coding agent's instructions. Your CLAUDE.md, skills and hooks now decide how
your code gets written, and nothing tests them. Claude Code ships about 25 releases a month, the
model behind an alias changes without a changelog, and a teammate's one-line edit to a skill can
switch it off. You find out weeks later, from the code.

config-drift-checker turns those instructions into eval cases and runs them on every pull request
and every Claude Code release, against a pinned baseline. When something stops working, you get a
red check that says what broke, why, and what moved, and it can propose the fix.

What it found when we ran it on ourselves

We run it on our own Claude Code setup every day, in public. Four things it caught that nothing
else would have:

- A one-line PR silently switched off a skill. We rewrote a skill's trigger description the
  way a careless PR would. The suite fell from 1.00 to 0.56, and the report named the cause by
  itself: the skill was *discovered but never invoked*, so fix the wording, not the packaging. The
  repair skill then fixed it and proved it with a green re-run, for $0.28.
  The report ·
  the repair
- Claude Code 2.1.310 stopped running shell-based evals on GitHub's Ubuntu runners, because
  of a sandbox check. Our release canary caught it the day it shipped; the Action now handles it.
- Our own tests were weaker than they looked. We sabotaged our setup 15 ways on purpose and the
  suite caught 6 (first run).
  Running the same cases with no plugin showed why: the model already did constructor injection
  3 times out of 3, so two of our rules steered nothing. We cut them, kept their checks as
  tripwires for the next model, and fixed the weak cases.
  The re-run sorts every
  miss into "the model does…