Agent Evaluation
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
Install
npx skills add lucaspmarie-a11y/claude-skills-vault --skill agent-evaluationgit clone --depth 1 https://github.com/lucaspmarie-a11y/claude-skills-vault.git /tmp/claude-skills-vault
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills-vault/plugins/antigravity-awesome-skills-claude/skills/agent-evaluation ~/.claude/skills/agent-evaluationPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Agent Evaluation You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer. You've built evaluation frameworks that catch issues before production: behavioral regression tests, capability assessments, and reliability metrics. You understand that the goal isn't 100% test pass rate—it ## Capabilities - agent-testing - benchmark-design - capability-assessment - reliability-metrics - regression-testing ## Requirements - testing-fundamentals - llm-fundamentals ## Patterns ### Statistical Test Evaluation Run tests multiple times and analyze result distributions ### Behavioral Contract Testing Define and test agent behavioral invariants ### Adversarial Testing Actively try to break agent behavior ## Anti-Patterns ### ❌ Single-Run Testing ### ❌ Only Happy Path Tests ### ❌ Output String Matching ## ⚠️ Sharp Edges | Issue | Severity | Solution | |-------|----------|----------| | Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation | | Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation | | Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
Details
- Skill name
agent-evaluation- Repository
- lucaspmarie-a11y/claude-skills-vault
- Path
plugins/antigravity-awesome-skills-claude/skills/agent-evaluation/SKILL.md- License
- MIT
- Repo updated
- 6 June 2026
FAQ
What does the Agent Evaluation skill do?
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
How do I install the Agent Evaluation skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add lucaspmarie-a11y/claude-skills-vault --skill agent-evaluation. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Agent Evaluation popular?
It lives in lucaspmarie-a11y/claude-skills-vault, which has 5 GitHub stars and 0 forks.
Related skills
Apify Influencer Discovery
5lucaspmarie-a11y/claude-skills-vault
Find and evaluate influencers for brand partnerships, verify authenticity, and track collaboration performance across Instagram, Facebook, YouTube, and TikTok.
00 Andruia Consultant
5lucaspmarie-a11y/claude-skills-vault
Arquitecto de Soluciones Principal y Consultor Tecnológico de Andru.ia. Diagnostica y traza la hoja de ruta óptima para proyectos de IA en español.
10 Andruia Skill Smith
5lucaspmarie-a11y/claude-skills-vault
Ingeniero de Sistemas de Andru.ia. Diseña, redacta y despliega nuevas habilidades (skills) dentro del repositorio siguiendo el Estándar de Diamante.
Agent Tool Builder
5lucaspmarie-a11y/claude-skills-vault
You are an expert in the interface between LLMs and the outside world. You've seen tools that work beautifully and tools that cause agents to hallucinate, loop, or fail silently. The difference is almost always in the design, not the implementation.
Advanced Evaluation
5lucaspmarie-a11y/claude-skills-vault
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
Agentmail
5lucaspmarie-a11y/claude-skills-vault
Email infrastructure for AI agents. Create accounts, send/receive emails, manage webhooks, and check karma balance via the AgentMail API.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.