Agent Evaluation Designer
Use this skill whenever the user wants to evaluate, test, or validate an AI agent, decide whether an agent is ready to ship or go live, choose how to grade an agent's answers (exact match, similarity, meaning, keywords, quality, or custom), design a test set of questions and expected answers, or interpret evaluation results into a go/no-go decision. Invoke it before the user hand-builds tests or declares an agent "done.
Install
npx skills add microsoft/cat-agent-skills --skill agent-evaluation-designergit clone --depth 1 https://github.com/microsoft/cat-agent-skills.git /tmp/cat-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/cat-agent-skills/submissions/agent-evaluation-designer ~/.claude/skills/agent-evaluation-designerPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Agent Evaluation Designer
You help the user design and run a **rigorous, defensible evaluation** of an AI
agent and turn the results into a clear **go / no-go** decision. Evaluation is a
product discipline, not a technical formality: your job is to make the user
define what "good" means *before* testing, pick the right way to measure it, and
stay accountable to the result.
Work through the five stages below in order. Do not skip stage 1 - most bad
evaluations fail because "good" was never defined. Ask concise questions when you
lack the information a stage needs; otherwise proceed and state your assumptions.
## Stage 1 - Define what "good" means
Establish the evaluation's purpose before writing a single test.
1. Ask what decision the evaluation must support (ship / don't ship, compare two
versions, catch regressions, satisfy a stakeholder or compliance gate).
2. Ask who the agent serves and the top real-world tasks it must get right.
3. For each task, define the **quality dimensions** that matter, choosing from:
- **Correctness / groundedness** - is the answer factually right and grounded
in the agent's sources?
- **Completeness** - does it cover the required points?
- **Relevance** - does it answer what was asked?
- **Tone / format / compliance** - does it meet wording, safety, or policy rules?
- **Tool / action use** - did it call the right capability or resource?Details
- Skill name
agent-evaluation-designer- Repository
- microsoft/cat-agent-skills
- Path
submissions/agent-evaluation-designer/SKILL.md- License
- MIT
- Repo updated
- 15 November 2026
FAQ
What does the Agent Evaluation Designer skill do?
Use this skill whenever the user wants to evaluate, test, or validate an AI agent, decide whether an agent is ready to ship or go live, choose how to grade an agent's answers (exact match, similarity, meaning, keywords, quality, or custom), design a test set of questions and expected answers, or interpret evaluation results into a go/no-go decision. Invoke it before the user hand-builds tests or declares an agent "done.
How do I install the Agent Evaluation Designer skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add microsoft/cat-agent-skills --skill agent-evaluation-designer. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Agent Evaluation Designer popular?
It lives in microsoft/cat-agent-skills, which has 110 GitHub stars and 122 forks.
Related skills
3 Statements
110microsoft/cat-agent-skills
Completes existing income statement, balance sheet and cash flow templates while preserving layout, grounding historical inputs and linking forecast formulas with reconciliation checks. Use for "fill in a 3-statement template", "complete an IS/BS/CF framework", "populate my integrated financial model", "link statements in this template", or "finish the forecasts in my three-statement workbook". Do NOT use for blank-slate models or general workbook edits (use xlsx), DCF valuation (use dcf-valuation), or buyout modeling (use lbo-model).
Accessibility Pass
110microsoft/cat-agent-skills
Use this skill whenever the user asks to check, review, or fix the accessibility of a PowerPoint deck, Word document, HTML page, or Markdown file, and before handing over any deck or document this agent just generated, so it does not ship with missing alt text, untitled slides, or unreadable colour contrast.
Accessible Learning Materials
110microsoft/cat-agent-skills
Maintains instructionally equivalent, accessible variants of a set of learning materials — lessons, readings, slides, and handouts — with parity checks against the original learning goals and recorded approvals. Use when the user asks to "make these materials accessible", "audit this lesson set for accessibility", "create an accessible version of this reading", "check reading level and alt text", or "keep my accessible variants in sync". Do NOT use for individual student accommodation or support-plan decisions (IEP, EHCP, or local equivalent), and do NOT use for generic document formatting (us
Agent Harness Explorer
110microsoft/cat-agent-skills
Use this skill whenever the user wants to inspect, understand, snapshot, compare, or document the capabilities of the current agent harness — for example "what can this harness do?", "which Python libraries are installed?", "what should I use to create Word documents?", "capture/remember a snapshot", "compare with my baseline", or "what changed since last week?". Prefer this skill BEFORE guessing what the runtime supports.
Agent Performance Triage
110microsoft/cat-agent-skills
Diagnose a live Microsoft Copilot Studio agent from its analytics and transcripts, find why it is underperforming, and produce a prioritized improvement backlog tied to specific topics, knowledge sources and tools. Use when an agent is already in production and someone asks why resolution or engagement is low, why users escalate or abandon, what the top unanswered questions are, how to improve an agent's containment or CSAT, or wants a post-launch review, health check or optimization sprint for a deployed agent.
Tech Prompt Engineering
248asgard-ai-platform/skills
Debug and harden production LLM prompts — handle prompt injection, output format drift, instruction forgetting in long contexts, and cross-model portability issues. Use this skill when the user ships an LLM-powered feature to production and needs to diagnose why outputs are inconsistent, unsafe, or regressed after model updates — NOT for basic 'write a better prompt' questions.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.