AI Prompt Regression Testing

Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.

9 stars 4 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add huytran1120/vietnam-agentic-skills --skill ai-prompt-regression-testing
Manual install
git clone --depth 1 https://github.com/huytran1120/vietnam-agentic-skills.git /tmp/vietnam-agentic-skills
mkdir -p ~/.claude/skills
cp -r /tmp/vietnam-agentic-skills/skills/ai-prompt-regression-testing ~/.claude/skills/ai-prompt-regression-testing

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# AI Prompt Regression Testing

## When to Use

- Compare a prompt change with an observed baseline on a fixed evaluation set.
- Detect schema, factuality, refusal, tool-selection or task-completion regressions.
- Record a reviewable decision when results are noisy, missing or contradictory.

## Workflow

1. Define the task, representative cases and forbidden outcomes. Use synthetic
   or approved/redacted data; freeze case IDs and the dataset revision before runs.
2. Record baseline/candidate prompt digests, exact available model/version,
   inference settings, tool versions and evaluation environment. Do not hardcode
   old model-family options or silently change models between comparisons.
3. Predeclare checks and acceptance rules. Use deterministic schema/field checks
   for structured output, behavioral checks for tools and a clear human/rubric
   standard for open text. Similarity does not prove factual or code correctness.
4. Run matched baseline/candidate cases under the same permissions and budget.
   Record latency/cost and repeated runs when variability matters. A timeout,
   missing output or judge error is incomplete, not a pass or a zero-cost result.
5. Treat judge outputs as evidence with limitations. Blind/randomize presentation
   where feasible, calibrate against human-labeled examples, and record judge
   identity/rubric. Do not ask a model to verify private reasoning or accept a
   judge's confidence as an objective correctness measurement.
Read the full skill on GitHub

Details

Skill name
ai-prompt-regression-testing
Path
skills/ai-prompt-regression-testing/SKILL.md
License
MIT
Repo updated
29 October 2026
Copies
Found in 2 GitHub repos

FAQ

What does the AI Prompt Regression Testing skill do?

Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.

How do I install the AI Prompt Regression Testing skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add huytran1120/vietnam-agentic-skills --skill ai-prompt-regression-testing. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is AI Prompt Regression Testing popular?

It lives in huytran1120/vietnam-agentic-skills, which has 9 GitHub stars and 4 forks.

Related skills

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.