Agent Evals

Build and maintain evaluation suites for AI agents: bootstrap 20-50 tasks from real failures, pick code-based vs model-based vs human graders by agent architecture (coding, conversational, research, computer-use), write unambiguous tasks with reference solutions, and keep suites alive via ownership and transcript reading. Use when creating evals from zero, adding graders or eval cases, debugging suspicious eval scores, or deciding pass@k vs pass^k metrics for an agent product.

10 stars 2 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add Archive228/lab-skills --skill agent-evals
Manual install
git clone --depth 1 https://github.com/Archive228/lab-skills.git /tmp/lab-skills
mkdir -p ~/.claude/skills
cp -r /tmp/lab-skills/skills/agent-evals ~/.claude/skills/agent-evals

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Agent Evals: Bootstrap, Grader Design, and Maintenance

This skill turns "the agent feels worse" into a measurable, fixable signal. It gives you an operating procedure for building evals for AI agents: how to bootstrap a first suite from real failures, how to choose graders (code-based, model-based, human) for each agent architecture, how to write tasks that grade fairly, which consistency metric to report, and how to keep the suite trustworthy over time. Distilled from Anthropic's engineering guide "Demystifying evals for AI agents."

## When to use

Use this skill when:
- You are asked to create an eval suite for an agent (coding, conversational/support, research, or computer-use) and none exists.
- You are adding eval cases, graders, or a CI eval gate to an existing agent project.
- An eval score looks suspicious (0% pass rate, sudden jump/drop, plateau) and you must diagnose whether the agent or the eval is at fault.
- You must choose or explain metrics (pass@k vs pass^k) or decide when a capability eval should become a regression eval.
- A model upgrade is planned and the team needs to verify it quickly.

Do NOT use this skill for: single-turn prompt/response benchmarking with no tools or state (standard eval practice suffices), production A/B test design, or training-data curation. If the team already has a healthy suite and just wants a new task added, jump straight to Rules 6-9.

## Rules

### Framing and vocabulary
Read the full skill on GitHub

Details

Skill name
agent-evals
Path
skills/agent-evals/SKILL.md
License
MIT
Repo updated
12 August 2026

FAQ

What does the Agent Evals skill do?

Build and maintain evaluation suites for AI agents: bootstrap 20-50 tasks from real failures, pick code-based vs model-based vs human graders by agent architecture (coding, conversational, research, computer-use), write unambiguous tasks with reference solutions, and keep suites alive via ownership and transcript reading. Use when creating evals from zero, adding graders or eval cases, debugging suspicious eval scores, or deciding pass@k vs pass^k metrics for an agent product.

How do I install the Agent Evals skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add Archive228/lab-skills --skill agent-evals. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Agent Evals popular?

It lives in Archive228/lab-skills, which has 10 GitHub stars and 2 forks.

Related skills

Agent Design Fundamentals

10

Archive228/lab-skills

Decision rules for whether to build an LLM agent and how to architect it, distilled from OpenAI's "A Practical Guide to Building Agents": build/no-build criteria, model selection, tool design, single-agent vs multi-agent orchestration (manager pattern vs decentralized handoffs), layered guardrails, and human-in-the-loop triggers. Use when designing, reviewing, or refactoring an agent system — before choosing an architecture, adding more agents, defining tools, or shipping an agent to production without guardrails.

AI, LLMs & Agents

Agent Failure Preflight

10

Archive228/lab-skills

Pre-deployment failure-mode gate for agentic AI systems, distilled from Microsoft AI Red Team's Taxonomy of Failure Modes in Agentic AI Systems v2.0 (grounded in 12 months of red-team engagements). Run it before shipping or materially changing an agent — adding tools/MCP servers/plugins, enabling persistent memory, adding sub-agents or orchestration, enabling computer use, or expanding permissions — to verify concrete controls against memory poisoning, HitL bypass, goal hijacking, supply-chain compromise, session context contamination, capability disclosure, and multi-agent cascade failures.

AI, LLMs & Agents

Agent Improvement Loop

10

Archive228/lab-skills

Operate a trace-driven improvement loop for a deployed agent: capture execution traces, convert human and model feedback into Promptfoo evals, gate the current harness against them, rank harness changes with HALO, and hand off a Codex-ready change set. Use when an agent is live and you need a repeatable cadence for diagnosing failures and shipping harness fixes — not for pre-launch prompt tinkering without traces.

AI, LLMs & Agents

Agent Builder

5

NikolasMarkou/agent-builder

Build, review, troubleshoot, optimize, and extend AI agents — single-agent, multi-agent, RAG, and production systems. Covers framework selection, pattern selection (topology, behavioral, data flow), agent orchestration, state design, and tool design. Defaults to Python with LangChain/LangGraph; also supports CrewAI, Strands, OpenAI Agents SDK, Google ADK, and Mastra. Use whenever the user wants to build, design, scaffold, review, debug, optimize, extend, or migrate an agent or agentic system.

AI, LLMs & Agents

Agent Sandboxing

10

Archive228/lab-skills

Containment architecture for agent products that execute code, browse, or call tools, distilled from Anthropic's "How we contain Claude across products". Use when designing or reviewing sandboxing, isolation, permission systems, egress controls, or untrusted-content handling for an agent, or when choosing between container, OS-sandbox, and VM isolation patterns; provides a threat-model checklist covering user misuse, model misbehavior, and external attacks.

AI, LLMs & Agents

Storyblok Use MCP

9

storyblok/agent-plugin

Use when interacting with Storyblok through the MCP server — the tool model, operation names and payload shapes for stories, assets, components, datasources and every other resource. Load it before calling `mcp__storyblok__` tools, alongside the skill that owns the task.

AI, LLMs & Agents

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.