Prompt Regression Suite

Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case pass criteria, CI gate thresholds, and a triage protocol for failures. For designing first-time evaluation of a new feature use ai-eval-plan instead.

1,573 stars 281 forksAI, LLMs & Agents Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add mohitagw15856/pm-claude-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add mohitagw15856/pm-claude-skills --skill prompt-regression-suite
Manual install
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git /tmp/pm-claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/pm-claude-skills/plugins/pm-agentops/skills/prompt-regression-suite ~/.claude/skills/prompt-regression-suite

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Prompt Regression Suite Skill

Every prompt tweak, model upgrade, and context change is a deploy. This skill designs the suite that runs on each one and answers a single question: *did anything that used to work stop working?*

## What This Skill Produces

- A **golden case set**: curated inputs with per-case pass criteria
- **Scoring methods** per case class (exact, rubric-judge, property checks)
- **CI gate thresholds** — what blocks a merge vs. what warns
- A **failure triage protocol** — flaky vs. regressed vs. golden-set-wrong

## Required Inputs

Ask for (if not already provided):
- **The feature and its contract** — what the LLM step receives and must produce
- **What has broken before** (or nearly) — past incidents seed the best cases
- **Real traffic examples** — 10-20 representative inputs, including ugly ones
- **What triggers a run** — prompt edits, model bumps, retrieval changes, all of the above?

## Building the Golden Set

Compose the set from four deliberate classes — not a random sample:

| Class | Purpose | Share |
|---|---|---|
| **Core paths** | The 5-10 inputs that represent most real traffic | ~40% |
| **Past failures** | Every input that caused a bug, complaint, or incident — permanently | ~25% |
| **Edge & adversarial** | Empty/huge inputs, wrong language, injection attempts, off-topic | ~25% |
| **Canaries** | Cases pinned to behaviours you never want to change (refusals, format, tone) | ~10% |
Read the full skill on GitHub

Details

Skill name
prompt-regression-suite
Path
plugins/pm-agentops/skills/prompt-regression-suite/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the Prompt Regression Suite skill do?

Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case pass criteria, CI gate thresholds, and a triage protocol for failures. For designing first-time evaluation of a new feature use ai-eval-plan instead.

How do I install the Prompt Regression Suite skill?

Claude Code plugin: /plugin marketplace add mohitagw15856/pm-claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Prompt Regression Suite popular?

It lives in mohitagw15856/pm-claude-skills, which has 1,573 GitHub stars and 281 forks, and is included in 1 curated skill list.

Related skills

Ab Test Planner

1.6k

mohitagw15856/pm-claude-skills

Diseña pruebas A/B estadísticamente rigurosas para características de producto, cambios de UI, flujos de onboarding y experimentos de precios. Úsalo cuando te pidan configurar un experimento, diseñar una prueba A/B, calcular tamaño de muestra o interpretar resultados de pruebas. Produce un plan de prueba completo con hipótesis, definición de variantes, tamaño de muestra, estimación de duración, métricas de seguridad y una guía de interpretación de resultados.

Testing & QAIn 1 list281

Ab Test Planner

1.6k

mohitagw15856/pm-claude-skills

प्रोडक्ट फ़ीचर, UI बदलाव, ऑनबोर्डिंग फ़्लो और प्राइसिंग प्रयोगों के लिए सांख्यिकीय रूप से कठोर A/B टेस्ट डिज़ाइन करें। जब कोई प्रयोग सेट करने, A/B टेस्ट डिज़ाइन करने, सैंपल साइज़ निकालने या टेस्ट के नतीजे समझाने को कहे, तब इसका उपयोग करें। परिकल्पना, वेरिएंट की परिभाषा, सैंपल साइज़, अवधि का अनुमान, गार्डरेल मेट्रिक और नतीजों की व्याख्या की गाइड के साथ एक पूरा टेस्ट प्लान तैयार करता है।

Testing & QAIn 1 list281

Accessible Travel Planner

1.6k

mohitagw15856/pm-claude-skills

Summarise what Accessible Travel Planner does in one line. Use when asked to [trigger phrases the user would say]. Produces [the concrete artifact].

Frontend & DesignIn 1 list281

RAG Architecture Review

1.6k

mohitagw15856/pm-claude-skills

Review an existing Retrieval-Augmented Generation system and find why it underperforms. Use when asked to review or audit a RAG pipeline, diagnose wrong/ungrounded answers from a 'chat with your docs' feature, or improve an already-built knowledge assistant. Produces a staged review — ingestion, chunking, retrieval, reranking, generation, evaluation — with prioritised findings, root causes, and concrete fixes.

AI, LLMs & AgentsIn 1 list281

Prompt Optimizer

1.6k

mohitagw15856/pm-claude-skills

Diagnose and rewrite an underperforming LLM prompt so it produces reliable, well-structured output. Use when asked to improve a prompt, fix a prompt that gives inconsistent or wrong results, reduce hallucination/refusals, or make output follow a format. Produces a rewritten prompt with a diagnosis of what was failing, the specific changes and why, and a small test set to verify the fix.

AI, LLMs & AgentsIn 1 list281

RAG Design Doc

1.6k

mohitagw15856/pm-claude-skills

Design a Retrieval-Augmented Generation system end to end. Use when asked to design a RAG pipeline, a 'chat with your docs' feature, a knowledge assistant, or to debug why a RAG system gives wrong/ungrounded answers. Produces a RAG design doc — ingestion & chunking, embeddings & index, retrieval & reranking, the generation prompt, grounding/citations, evaluation, and failure modes with mitigations.

AI, LLMs & AgentsIn 1 list281

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.