Skill Forge Eval

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns executor, grader, comparator, and analyzer sub-agents for parallel evaluation. Generates eval_metadata.json, grading.json, and feedback reports. Use when user says "eval skill", "test skill", "run evals", "evaluate skill", "skill evals", "test skill quality", "run skill tests", or "skill evaluation".

189 stars 44 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval
Manual install
git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git /tmp/skill-forge
mkdir -p ~/.claude/skills
cp -r /tmp/skill-forge/skills/skill-forge-eval ~/.claude/skills/skill-forge-eval

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Skill Evaluation Pipeline

Run structured evaluations against Claude Code skills to verify triggering,
correctness, and quality using a multi-agent pipeline.

## Process

### Step 1: Define Eval Set

Accept eval definitions from:
- **Path to eval set JSON**: `evals/evals.json` or user-specified file
- **Inline prompts**: User provides eval queries directly
- **Auto-generated**: Generate from skill description (see Step 1b)

**Eval set JSON schema:**
```json
{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "evals": [
    {
      "eval_id": 0,
      "eval_name": "descriptive-name",
      "prompt": "The user's task prompt",
      "input_files": [],
      "assertions": [
        {
          "name": "output-has-score",
          "check": "Output contains a numeric score between 0-100",
          "weight": 1.0
        }
      ],
      "should_trigger": true
    }
  ]
}
```

#### Step 1b: Auto-Generate Eval Set

If no eval set exists, generate one:
1. Read the skill's SKILL.md description and instructions
2. Run `python scripts/generate_eval_set.py <skill-path>` to produce a starter set
3. Present the generated set to the user for review and editing
4. User approves or modifies before proceeding

### Step 2: Set Up Workspace

Create the eval workspace **outside** the skill directory to avoid confusing eval
artifacts with skill files. Use a sibling directory or a dedicated location:
Read the full skill on GitHub

Details

Skill name
skill-forge-eval
Path
skills/skill-forge-eval/SKILL.md
License
MIT
Repo updated
10 April 2026

FAQ

What does the Skill Forge Eval skill do?

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns executor, grader, comparator, and analyzer sub-agents for parallel evaluation. Generates eval_metadata.json, grading.json, and feedback reports. Use when user says "eval skill", "test skill", "run evals", "evaluate skill", "skill evals", "test skill quality", "run skill tests", or "skill evaluation".

How do I install the Skill Forge Eval skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Skill Forge Eval popular?

It lives in AgriciDaniel/skill-forge, which has 189 GitHub stars and 44 forks.

Related skills

Skill Forge

189

AgriciDaniel/skill-forge

Ultimate Claude Code skill creator and architect. Designs, scaffolds, builds, reviews, evolves, and publishes production-grade Claude Code skills following the Agent Skills open standard and 3-layer architecture (directive, orchestration, execution). Handles single-file skills, multi-skill orchestrators with sub-skills and subagents, MCP-enhanced workflows, and full skill ecosystems. Industry detection for skill domain. Triggers on: "create skill", "build skill", "new skill", "skill creator", "skill builder", "skill-forge", "design skill", "scaffold skill", "review skill", "improve skill", "pu

Skill & Plugin Authoring44

Skill Forge Benchmark

189

AgriciDaniel/skill-forge

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations. Runs multiple trials per eval for statistical reliability, aggregates results into benchmark.json, and generates comparison reports between skill versions. Use when user says "benchmark skill", "measure skill performance", "skill metrics", "compare skill versions", "skill performance", "track skill improvement", "skill regression test", or "skill A/B test".

Testing & QA44

Skill Forge Build

189

AgriciDaniel/skill-forge

Scaffold and build Claude Code skills from plans or descriptions. Generates SKILL.md files, sub-skills, scripts, references, agents, and templates following the Agent Skills standard. Use when user says "build skill", "scaffold skill", "generate skill", "create SKILL.md", or "implement skill".

AI, LLMs & Agents44

Skill Forge Publish

189

AgriciDaniel/skill-forge

Package and distribute Claude Code skills for sharing via GitHub, Claude.ai uploads, or team deployment. Creates install scripts, documentation, and .skill packages. Use when user says "publish skill", "share skill", "package skill", "distribute skill", or "release skill".

AI, LLMs & Agents44

Skill Forge Convert

189

AgriciDaniel/skill-forge

Convert Claude Code skills to work on OpenAI Codex, Google Gemini CLI, Google Antigravity, and Cursor. Analyzes platform-specific features, generates target files (openai.yaml, AGENTS.md, GEMINI.md, .mdc rules), adapts frontmatter, converts MCP config, and produces compatibility reports. Use when user says "convert skill", "port skill", "multi-platform", "skill for codex", "skill for gemini", "skill for antigravity", "skill for cursor", "cross-platform skill", "convert to codex", "convert to gemini", "convert to antigravity", or "convert to cursor".

AI, LLMs & Agents44

Skill Forge Review

189

AgriciDaniel/skill-forge

Audit and validate existing Claude Code skills for quality, triggering accuracy, structure compliance, and best practices. Scores skills on a 0-100 scale and provides prioritized improvement recommendations. Use when user says "review skill", "audit skill", "check skill", "validate skill", or "skill quality".

AI, LLMs & Agents44

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.