Agent Evaluation Engineering

Design reproducible evaluations for agent capabilities, trust, safety, trajectories, regressions, latency, and cost.

9 stars 3 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add sandbaseai/workbuddy-skill --skill agent-evaluation-engineering
Manual install
git clone --depth 1 https://github.com/sandbaseai/workbuddy-skill.git /tmp/workbuddy-skill
mkdir -p ~/.claude/skills
cp -r /tmp/workbuddy-skill/skills/agent-evaluation-engineering ~/.claude/skills/agent-evaluation-engineering

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# AI Agent Evaluation Engineering

Use this skill when creating or reviewing an evaluation suite for an AI agent. Treat an evaluation as an executable product contract, not a collection of impressive demos.

## 1. Define the contract and risk

Write down the user goal, allowed inputs, observable outputs, available tools, permissions, side effects, latency and cost limits, and acceptable escalation behavior. Rank failure modes by impact and likelihood. Keep capability quality separate from trust and safety: an agent can complete a task and still leak data, exceed authority, or make an irreversible change.

## 2. Build representative cases

Create cases from real user journeys, production failures, domain edge cases, ambiguous requests, missing context, tool outages, permission boundaries, prompt injection, and unsafe requests. Include single-turn and multi-turn cases where memory, correction, or recovery matters.

Partition cases into:

- development examples for rapid iteration;
- a held-out regression set that prompts and graders do not see during tuning;
- adversarial cases focused on authority, privacy, injection, and irreversible effects.

Turn every confirmed production failure into a minimized regression case. Remove duplicates and document why each case exists.

## 3. Specify evidence and graders

For each case, define the expected invariant before running it. Prefer observable evidence:

- final answer facts and citations;
Read the full skill on GitHub

Details

Skill name
agent-evaluation-engineering
Path
skills/agent-evaluation-engineering/SKILL.md
License
MIT
Repo updated
8 September 2026

FAQ

What does the Agent Evaluation Engineering skill do?

Design reproducible evaluations for agent capabilities, trust, safety, trajectories, regressions, latency, and cost.

How do I install the Agent Evaluation Engineering skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add sandbaseai/workbuddy-skill --skill agent-evaluation-engineering. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Agent Evaluation Engineering popular?

It lives in sandbaseai/workbuddy-skill, which has 9 GitHub stars and 3 forks.

Related skills

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.