LLM Evaluation
Use when evaluating LLM or RAG applications, comparing models/prompts, detecting regressions, designing benchmarks, or validating safety and quality improvements.
Install
npx skills add sandbaseai/workbuddy-skill --skill llm-evaluationgit clone --depth 1 https://github.com/sandbaseai/workbuddy-skill.git /tmp/workbuddy-skill
mkdir -p ~/.claude/skills
cp -r /tmp/workbuddy-skill/skills/llm-evaluation ~/.claude/skills/llm-evaluationPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# LLM Evaluation 把“看起来更好”变成可复现、可解释、能发现回归的证据。评测应围绕任务风险和用户目标组合自动指标、人工判断、对抗样本、线上信号与成本/延迟,而不是追求一个脱离场景的总分。 ## 使用边界 - 开始前锁定模型/provider、版本、系统提示、工具、检索器、参数、数据快照、随机性、语言、时区和评测 owner。 - 默认使用合成、公开或脱敏样本;不得将真实用户输入、反馈或完整模型输出写入长期评测集,除非有明确授权、最小化和删除路径。 - 不擅自执行生产 A/B、收集用户画像、切换模型、改变安全策略或发布评测结论;这些操作需要独立授权。 - prompt、context、答案、grader 输出和 trace 可能含敏感信息;报告使用哈希、聚合和最小引用,保留失败样本需单独审批。 ## 评测契约与数据集 先定义任务、受众、成功标准、不可接受错误、拒答条件、SLO、成本上限和版本范围。数据集按真实分布分层:常见、边界、长上下文、多语言、无答案、歧义、工具失败、权限、注入和安全样本;保留训练/评测隔离,避免泄漏。 每个 case 记录稳定 ID、输入类别、期望行为/参考答案(可选)、允许证据、风险等级、数据版本和评分规则。对随机模型运行固定 seed 或多次重复,报告样本量、缺失、置信区间和异常,而非只报告均值。 ## 指标组合 - **任务结果**:分类用 precision/recall/F1 与混淆矩阵;结构化输出验证 schema、类型、必填字段和业务不变量。 - **文本质量**:BLEU/ROUGE 等重叠指标只适合相应任务;事实性、相关性、完整性和可读性需要语义或人工评审。 - **RAG**:分别测 retrieval Recall@k/MRR/nDCG、context relevance、claim groundedness、citation correctness 和无依据拒答。 - **安全与可靠性**:越权、提示注入、敏感信息泄露、危险建议、工具滥用、超时、重试和 graceful refusal 必须作为独立门禁。 - **运营**:p50/p95/p99 延迟、token、错误率、缓存命中、provider 成本和可用性与质量一起看。 不要把不同风险、不同任务或不同语言简单平均;按预先声明的权重和分层报告,保留逐 case 结果。 ## 人工评审与 LLM-as-Judge 评分 rubric 写出 0/1/2 或等级的可观察锚点、必错条件和“不确定/不适用”。人工评审使用盲评、随机顺序、双评审和分歧仲裁,计算一致性并抽查 grader 漂移。pairwise 比较需平衡顺序和位置偏差。 LLM judge 只能作为可校准的测量工具:固定 judge 版本/提示,提供 rubric 与必要证据,先用人工金标准校准偏差,检查长度、风格、位置和自我偏好。judge 分数不能单独证明事实、安全或生产改进;关键样本必须人工复核。
Details
- Skill name
llm-evaluation- Repository
- sandbaseai/workbuddy-skill
- Path
skills/llm-evaluation/SKILL.md- License
- MIT
- Repo updated
- 8 September 2026
FAQ
What does the LLM Evaluation skill do?
Use when evaluating LLM or RAG applications, comparing models/prompts, detecting regressions, designing benchmarks, or validating safety and quality improvements.
How do I install the LLM Evaluation skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add sandbaseai/workbuddy-skill --skill llm-evaluation. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is LLM Evaluation popular?
It lives in sandbaseai/workbuddy-skill, which has 9 GitHub stars and 3 forks.
Related skills
Access Control Security Best Practices
9sandbaseai/workbuddy-skill
Design, review, implement, and verify least-privilege access controls across users, services, tenants, resources, and administrative workflows.
Accessibility Review
9sandbaseai/workbuddy-skill
Use when reviewing a web interface or interaction for semantic, keyboard, focus, screen-reader, ARIA, and contrast accessibility risks.
Acquire Codebase Knowledge
9sandbaseai/workbuddy-skill
Use when the user asks to map, document, or onboard into an existing codebase and needs evidence-backed architecture, structure, conventions, integrations, testing, and concerns documentation.
MCP Builder
9sandbaseai/workbuddy-skill
Use when designing, implementing, or evaluating an MCP server that exposes an external service through reliable, discoverable, and testable tools.
LLM App Security
9sandbaseai/workbuddy-skill
Use when designing or reviewing an LLM-powered application for prompt injection, unsafe outputs, data leakage, tenant isolation, and abuse risks.
MCP CLI
9sandbaseai/workbuddy-skill
Use when discovering, inspecting, or calling MCP servers and tools from a command line, with explicit schema, target, permission, and side-effect checks.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.