Agent Evaluation Designer

Use this skill whenever the user wants to evaluate, test, or validate an AI agent, decide whether an agent is ready to ship or go live, choose how to grade an agent's answers (exact match, similarity, meaning, keywords, quality, or custom), design a test set of questions and expected answers, or interpret evaluation results into a go/no-go decision. Invoke it before the user hand-builds tests or declares an agent "done.

110 stars 122 forksAI, LLMs & AgentsOfficial

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add microsoft/cat-agent-skills --skill agent-evaluation-designer
Manual install
git clone --depth 1 https://github.com/microsoft/cat-agent-skills.git /tmp/cat-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/cat-agent-skills/submissions/agent-evaluation-designer ~/.claude/skills/agent-evaluation-designer

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Agent Evaluation Designer

You help the user design and run a **rigorous, defensible evaluation** of an AI
agent and turn the results into a clear **go / no-go** decision. Evaluation is a
product discipline, not a technical formality: your job is to make the user
define what "good" means *before* testing, pick the right way to measure it, and
stay accountable to the result.

Work through the five stages below in order. Do not skip stage 1 - most bad
evaluations fail because "good" was never defined. Ask concise questions when you
lack the information a stage needs; otherwise proceed and state your assumptions.

## Stage 1 - Define what "good" means

Establish the evaluation's purpose before writing a single test.

1. Ask what decision the evaluation must support (ship / don't ship, compare two
   versions, catch regressions, satisfy a stakeholder or compliance gate).
2. Ask who the agent serves and the top real-world tasks it must get right.
3. For each task, define the **quality dimensions** that matter, choosing from:
   - **Correctness / groundedness** - is the answer factually right and grounded
     in the agent's sources?
   - **Completeness** - does it cover the required points?
   - **Relevance** - does it answer what was asked?
   - **Tone / format / compliance** - does it meet wording, safety, or policy rules?
   - **Tool / action use** - did it call the right capability or resource?
Read the full skill on GitHub

Details

Skill name
agent-evaluation-designer
Path
submissions/agent-evaluation-designer/SKILL.md
License
MIT
Repo updated
15 November 2026

FAQ

What does the Agent Evaluation Designer skill do?

Use this skill whenever the user wants to evaluate, test, or validate an AI agent, decide whether an agent is ready to ship or go live, choose how to grade an agent's answers (exact match, similarity, meaning, keywords, quality, or custom), design a test set of questions and expected answers, or interpret evaluation results into a go/no-go decision. Invoke it before the user hand-builds tests or declares an agent "done.

How do I install the Agent Evaluation Designer skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add microsoft/cat-agent-skills --skill agent-evaluation-designer. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Agent Evaluation Designer popular?

It lives in microsoft/cat-agent-skills, which has 110 GitHub stars and 122 forks.

Related skills

3 Statements

110

microsoft/cat-agent-skills

Completes existing income statement, balance sheet and cash flow templates while preserving layout, grounding historical inputs and linking forecast formulas with reconciliation checks. Use for "fill in a 3-statement template", "complete an IS/BS/CF framework", "populate my integrated financial model", "link statements in this template", or "finish the forecasts in my three-statement workbook". Do NOT use for blank-slate models or general workbook edits (use xlsx), DCF valuation (use dcf-valuation), or buyout modeling (use lbo-model).

AI, LLMs & Agents122

Accessibility Pass

110

microsoft/cat-agent-skills

Use this skill whenever the user asks to check, review, or fix the accessibility of a PowerPoint deck, Word document, HTML page, or Markdown file, and before handing over any deck or document this agent just generated, so it does not ship with missing alt text, untitled slides, or unreadable colour contrast.

Frontend & Design122

Accessible Learning Materials

110

microsoft/cat-agent-skills

Maintains instructionally equivalent, accessible variants of a set of learning materials — lessons, readings, slides, and handouts — with parity checks against the original learning goals and recorded approvals. Use when the user asks to "make these materials accessible", "audit this lesson set for accessibility", "create an accessible version of this reading", "check reading level and alt text", or "keep my accessible variants in sync". Do NOT use for individual student accommodation or support-plan decisions (IEP, EHCP, or local equivalent), and do NOT use for generic document formatting (us

Learning & Education122

Agent Harness Explorer

110

microsoft/cat-agent-skills

Use this skill whenever the user wants to inspect, understand, snapshot, compare, or document the capabilities of the current agent harness — for example "what can this harness do?", "which Python libraries are installed?", "what should I use to create Word documents?", "capture/remember a snapshot", "compare with my baseline", or "what changed since last week?". Prefer this skill BEFORE guessing what the runtime supports.

AI, LLMs & Agents122

Agent Performance Triage

110

microsoft/cat-agent-skills

Diagnose a live Microsoft Copilot Studio agent from its analytics and transcripts, find why it is underperforming, and produce a prioritized improvement backlog tied to specific topics, knowledge sources and tools. Use when an agent is already in production and someone asks why resolution or engagement is low, why users escalate or abandon, what the top unanswered questions are, how to improve an agent's containment or CSAT, or wants a post-launch review, health check or optimization sprint for a deployed agent.

AI, LLMs & Agents122

Tech Prompt Engineering

248

asgard-ai-platform/skills

Debug and harden production LLM prompts — handle prompt injection, output format drift, instruction forgetting in long contexts, and cross-model portability issues. Use this skill when the user ships an LLM-powered feature to production and needs to diagnose why outputs are inconsistent, unsafe, or regressed after model updates — NOT for basic 'write a better prompt' questions.

AI, LLMs & Agents30

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.