Agent Evaluation

You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add lucaspmarie-a11y/claude-skills-vault --skill agent-evaluation
Manual install
git clone --depth 1 https://github.com/lucaspmarie-a11y/claude-skills-vault.git /tmp/claude-skills-vault
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills-vault/plugins/antigravity-awesome-skills-claude/skills/agent-evaluation ~/.claude/skills/agent-evaluation

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Agent Evaluation

You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in
production. You've learned that evaluating LLM agents is fundamentally different from
testing traditional software—the same input can produce different outputs, and "correct"
often has no single answer.

You've built evaluation frameworks that catch issues before production: behavioral regression
tests, capability assessments, and reliability metrics. You understand that the goal isn't
100% test pass rate—it

## Capabilities

- agent-testing
- benchmark-design
- capability-assessment
- reliability-metrics
- regression-testing

## Requirements

- testing-fundamentals
- llm-fundamentals

## Patterns

### Statistical Test Evaluation

Run tests multiple times and analyze result distributions

### Behavioral Contract Testing

Define and test agent behavioral invariants

### Adversarial Testing

Actively try to break agent behavior

## Anti-Patterns

### ❌ Single-Run Testing

### ❌ Only Happy Path Tests

### ❌ Output String Matching

## ⚠️ Sharp Edges

| Issue | Severity | Solution |
|-------|----------|----------|
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
Read the full skill on GitHub

Details

Skill name
agent-evaluation
Path
plugins/antigravity-awesome-skills-claude/skills/agent-evaluation/SKILL.md
License
MIT
Repo updated
6 June 2026

FAQ

What does the Agent Evaluation skill do?

You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.

How do I install the Agent Evaluation skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add lucaspmarie-a11y/claude-skills-vault --skill agent-evaluation. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Agent Evaluation popular?

It lives in lucaspmarie-a11y/claude-skills-vault, which has 5 GitHub stars and 0 forks.

Related skills

Apify Influencer Discovery

5

lucaspmarie-a11y/claude-skills-vault

Find and evaluate influencers for brand partnerships, verify authenticity, and track collaboration performance across Instagram, Facebook, YouTube, and TikTok.

Frontend & Design

00 Andruia Consultant

5

lucaspmarie-a11y/claude-skills-vault

Arquitecto de Soluciones Principal y Consultor Tecnológico de Andru.ia. Diagnostica y traza la hoja de ruta óptima para proyectos de IA en español.

Other

10 Andruia Skill Smith

5

lucaspmarie-a11y/claude-skills-vault

Ingeniero de Sistemas de Andru.ia. Diseña, redacta y despliega nuevas habilidades (skills) dentro del repositorio siguiendo el Estándar de Diamante.

Other

Agent Tool Builder

5

lucaspmarie-a11y/claude-skills-vault

You are an expert in the interface between LLMs and the outside world. You've seen tools that work beautifully and tools that cause agents to hallucinate, loop, or fail silently. The difference is almost always in the design, not the implementation.

AI, LLMs & Agents

Advanced Evaluation

5

lucaspmarie-a11y/claude-skills-vault

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

AI, LLMs & Agents

Agentmail

5

lucaspmarie-a11y/claude-skills-vault

Email infrastructure for AI agents. Create accounts, send/receive emails, manage webhooks, and check karma balance via the AgentMail API.

AI, LLMs & Agents

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.