Dspy Evaluation Suite

Use for evaluating DSPy programs with Evaluate, answer_exact_match, SemanticF1, custom metrics, baselines, and program comparisons.

125 stars 13 forksSoftware Development Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add OmidZamani/dspy-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add OmidZamani/dspy-skills --skill dspy-evaluation-suite
Manual install
git clone --depth 1 https://github.com/OmidZamani/dspy-skills.git /tmp/dspy-skills
mkdir -p ~/.claude/skills
cp -r /tmp/dspy-skills/skills/dspy-evaluation-suite ~/.claude/skills/dspy-evaluation-suite

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# DSPy Evaluation Suite

## Goal

Systematically evaluate DSPy programs using built-in and custom metrics with parallel execution.

## When to Use

- Measuring program performance before/after optimization
- Comparing different program variants
- Establishing baselines
- Validating production readiness

## Related Skills

- Use with any optimizer: [dspy-bootstrap-fewshot](../dspy-bootstrap-fewshot/SKILL.md), [dspy-miprov2-optimizer](../dspy-miprov2-optimizer/SKILL.md), [dspy-gepa-reflective](../dspy-gepa-reflective/SKILL.md)
- Evaluate RAG pipelines: [dspy-rag-pipeline](../dspy-rag-pipeline/SKILL.md)

## Inputs

| Input | Type | Description |
|-------|------|-------------|
| `program` | `dspy.Module` | Program to evaluate |
| `devset` | `list[dspy.Example]` | Evaluation examples |
| `metric` | `callable` | Scoring function |
| `num_threads` | `int` | Parallel threads |

## Outputs

| Output | Type | Description |
|--------|------|-------------|
| `score` | `float` | Average metric score |
| `results` | `list` | Per-example results |

## Workflow

### Phase 1: Setup Evaluator

```python
from dspy.evaluate import Evaluate

evaluator = Evaluate(
    devset=devset,
    metric=my_metric,
    num_threads=8,
    display_progress=True
)
```

### Phase 2: Run Evaluation
Read the full skill on GitHub

Details

Skill name
dspy-evaluation-suite
Path
skills/dspy-evaluation-suite/SKILL.md
License
MIT
Repo updated
23 June 2026

FAQ

What does the Dspy Evaluation Suite skill do?

Use for evaluating DSPy programs with Evaluate, answer_exact_match, SemanticF1, custom metrics, baselines, and program comparisons.

How do I install the Dspy Evaluation Suite skill?

Claude Code plugin: /plugin marketplace add OmidZamani/dspy-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Dspy Evaluation Suite popular?

It lives in OmidZamani/dspy-skills, which has 125 GitHub stars and 13 forks, and is included in 1 curated skill list.

Related skills

Browse the most popular Claude skills, all software development skills, or write your own with the skills generator.