AI Eval Plan
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
Install
/plugin marketplace add mohitagw15856/pm-claude-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plangit clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git /tmp/pm-claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/pm-claude-skills/plugins/pm-agentops/skills/ai-eval-plan ~/.claude/skills/ai-eval-planPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# AI Eval Plan Skill You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship. ## Required Inputs Ask for these only if they aren't already provided: - **The feature & task** — what the model does and what "good output" means to a user. - **Failure modes that matter** — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow). - **Available data** — any real examples, logs, or labelled cases; or note there are none yet. - **Who judges quality** — automated checks, an LLM judge, human raters, or a mix. - **The decision this gates** — ship/no-ship, model selection, or prompt iteration. ## Output Format ### Eval Plan: [feature] **1. What we're measuring** — the task, and a one-line definition of a good vs. bad response. **2. Eval dataset** - **Cases:** how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set). - **Coverage:** the slices/scenarios that must be represented (edge cases, adversarial, each major input type). - **Golden answers / references:** present or not, and how they were created. **3. Metrics & rubric** - **Per-dimension scores** — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes.
Details
- Skill name
ai-eval-plan- Repository
- mohitagw15856/pm-claude-skills
- Path
plugins/pm-agentops/skills/ai-eval-plan/SKILL.md- License
- MIT
- Repo updated
- 18 November 2026
- Listed in
- BehiSecc/awesome-claude-skills
FAQ
What does the AI Eval Plan skill do?
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
How do I install the AI Eval Plan skill?
Claude Code plugin: /plugin marketplace add mohitagw15856/pm-claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is AI Eval Plan popular?
It lives in mohitagw15856/pm-claude-skills, which has 1,573 GitHub stars and 281 forks, and is included in 1 curated skill list.
Related skills
Ab Test Planner
1.6kmohitagw15856/pm-claude-skills
Diseña pruebas A/B estadísticamente rigurosas para características de producto, cambios de UI, flujos de onboarding y experimentos de precios. Úsalo cuando te pidan configurar un experimento, diseñar una prueba A/B, calcular tamaño de muestra o interpretar resultados de pruebas. Produce un plan de prueba completo con hipótesis, definición de variantes, tamaño de muestra, estimación de duración, métricas de seguridad y una guía de interpretación de resultados.
Ab Test Planner
1.6kmohitagw15856/pm-claude-skills
प्रोडक्ट फ़ीचर, UI बदलाव, ऑनबोर्डिंग फ़्लो और प्राइसिंग प्रयोगों के लिए सांख्यिकीय रूप से कठोर A/B टेस्ट डिज़ाइन करें। जब कोई प्रयोग सेट करने, A/B टेस्ट डिज़ाइन करने, सैंपल साइज़ निकालने या टेस्ट के नतीजे समझाने को कहे, तब इसका उपयोग करें। परिकल्पना, वेरिएंट की परिभाषा, सैंपल साइज़, अवधि का अनुमान, गार्डरेल मेट्रिक और नतीजों की व्याख्या की गाइड के साथ एक पूरा टेस्ट प्लान तैयार करता है।
Accessible Travel Planner
1.6kmohitagw15856/pm-claude-skills
Summarise what Accessible Travel Planner does in one line. Use when asked to [trigger phrases the user would say]. Produces [the concrete artifact].
AI Feature Prd
1.6kmohitagw15856/pm-claude-skills
Write a PRD for an AI-powered feature, covering the things normal PRDs miss. Use when asked to spec an AI/LLM feature, write a PRD for a feature that uses a model, or plan an AI capability (assistant, summarizer, generator, classifier). Produces an AI feature PRD — problem & UX of uncertainty, model approach, eval criteria, guardrails, fallback behaviour, the data flywheel, and cost/latency budget.
AI Ethics Review
1.6kmohitagw15856/pm-claude-skills
Conduct a structured ethical review of an AI or ML feature, model, or product. Use when preparing to deploy an AI system, assessing algorithmic risk, auditing a model for bias, or producing a responsible AI impact assessment. Produces a structured ethics review covering fairness, transparency, privacy, safety, accountability, and societal impact with a risk tier score, pre-deployment checklist, and prioritised mitigations.
AI Output Verifier
1.6kmohitagw15856/pm-claude-skills
Check AI output before you trust or use it — where it's likely wrong, what to verify, and how to catch confident-sounding errors. Use when asked can I trust this AI answer, how do I verify what AI told me, fact-check this AI output, or is this AI response reliable. Produces a risk read on the specific output (the claims most likely to be wrong or made up), the parts that need independent verification vs the parts that are low-risk, how to actually verify each, the tells of AI hallucination and overconfidence, and a habit for building verification into your AI use — because AI is confidently wr
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.