M365 Agent Evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

3,222 stars 372 forksAI, LLMs & AgentsOfficial Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add microsoft/skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add microsoft/skills --skill m365-agent-evaluator
Manual install
git clone --depth 1 https://github.com/microsoft/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/.github/plugins/microsoft-365-agents-toolkit/skills/m365-agent-evaluator ~/.claude/skills/m365-agent-evaluator

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# M365 Agent Evaluator

Use this skill to help users evaluate Microsoft 365 Copilot declarative agents with `@microsoft/m365-copilot-eval`. The skill designs schema-compatible eval datasets, runs the public preview CLI, analyzes results, and recommends targeted fixes.

Default to Microsoft 365 Agents Toolkit (ATK) projects when detected, but do not hard-stop solely because the current directory is not ATK. The CLI can also evaluate deployed agents with an explicit `M365_AGENT_ID` or `--m365-agent-id`.

## Always use this CLI invocation

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals
```

Do not recommend the old private `aka.ms` installer, global installs, bare `runevals`, bare `npx runevals`, `--input`, or `--html`.

## Activation workflow

1. Identify the user goal: setup, dataset authoring, running evals, analyzing results, or updating an existing eval suite.
2. Load only the reference needed for the current goal:
   - `references/workflow.md` for the end-to-end operator workflow and CLI commands.
   - `references/azure-setup.md` for prerequisites, env files, and secret handling.
   - `references/eval-templates.md` when creating or editing eval datasets.
   - `references/pra-framework.md` when deciding what scenarios to generate.
   - `references/result-analysis.md` after JSON/CSV/HTML results exist.
   - `references/guardrails.md` before writing files, handling secrets, clearing cache, signing out, or troubleshooting.
3. Detect project shape:
Read the full skill on GitHub

Details

Skill name
m365-agent-evaluator
Path
.github/plugins/microsoft-365-agents-toolkit/skills/m365-agent-evaluator/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the M365 Agent Evaluator skill do?

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

How do I install the M365 Agent Evaluator skill?

Claude Code plugin: /plugin marketplace add microsoft/skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is M365 Agent Evaluator popular?

It lives in microsoft/skills, which has 3,222 GitHub stars and 372 forks, and is included in 2 curated skill lists.

Related skills

Azure AI Textanalytics Py

3.2k

microsoft/skills

Azure AI Text Analytics SDK for sentiment analysis, entity recognition, key phrases, language detection, PII, and healthcare NLP. Use for natural language processing on text. Triggers: "text analytics", "sentiment analysis", "entity recognition", "key phrase", "PII detection", "TextAnalyticsClient".

DevOps & CloudIn 2 lists372

Azure AI Anomalydetector Java

3.2k

microsoft/skills

Build anomaly detection applications with Azure AI Anomaly Detector SDK for Java. Use when implementing univariate/multivariate anomaly detection, time-series analysis, or AI-powered monitoring.

DevOps & CloudIn 2 lists372

Azure AI Openai Dotnet

3.2k

microsoft/skills

Azure OpenAI SDK for .NET. Client library for Azure OpenAI and OpenAI services. Use for chat completions, embeddings, image generation, audio transcription, and assistants. Triggers: "Azure OpenAI", "AzureOpenAIClient", "ChatClient", "chat completions .NET", "GPT-4", "embeddings", "DALL-E", "Whisper", "OpenAI .NET".

AI, LLMs & AgentsIn 2 lists372

M365 Agents Dotnet

3.2k

microsoft/skills

Microsoft 365 Agents SDK for .NET. Build multichannel agents for Teams/M365/Copilot Studio with ASP.NET Core hosting, AgentApplication routing, and MSAL-based auth. Triggers: "Microsoft 365 Agents SDK", "Microsoft.Agents", "AddAgentApplicationOptions", "AgentApplication", "AddAgentAspNetAuthentication", "Copilot Studio client", "IAgentHttpAdapter".

AI, LLMs & AgentsIn 2 lists372

Lieflat Less AI Tone

3.2k

larashero3-dotcom/writing-dna-skill

按 SKILL.md 明确列出的规则识别并改写写作中的 AI 痕迹。只能处理清单内的问题;未命中规则的文字必须原样保留,也不能改变文章框架。适用于写作完成后的成稿清理。 Remove AI writing tells using an explicit whitelist of rules; leaves unmatched text untouched.

AI, LLMs & Agents293

M365 Agents Py

3.2k

microsoft/skills

Microsoft 365 Agents SDK for Python. Build multichannel agents for Teams/M365/Copilot Studio with aiohttp hosting, AgentApplication routing, streaming responses, and MSAL-based auth. Triggers: "Microsoft 365 Agents SDK", "microsoft_agents", "AgentApplication", "start_agent_process", "TurnContext", "Copilot Studio client", "CloudAdapter".

AI, LLMs & AgentsIn 2 lists372

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.