Eval Runner MCP Server: Install & Setup

Install Eval Runner in Claude Code

Claude Code

claude mcp add eval-runner -e YOUR_API_KEY=... -- npx -y mcp-eval-runner

mcp.json (Claude Desktop, Cursor, VS Code, Windsurf)

{
  "mcpServers": {
    "eval-runner": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "..."
      }
    }
  }
}

Replace each ... with your own value.

Frequently asked questions

What does the Eval Runner MCP server do?

A standardized testing harness for MCP servers and agent workflows

Is the Eval Runner MCP server free?

The server is free, but you will likely need an account. The server itself is free, but it connects to a service that needs an account or API key, which may have paid tiers. Check the provider for current pricing.

Does Eval Runner run locally or remotely?

Local. Runs on your machine (stdio). You install it with npx, uvx, Docker or from source.

How do I add Eval Runner to Claude Code?

Run: claude mcp add eval-runner -e YOUR_API_KEY=... -- npx -y mcp-eval-runner

More ai agents & coding agents MCP servers

View all

Cua

31k

trycua/cua

MCP server for the Computer-Use Agent (CUA), allowing you to run CUA through Claude Desktop or other MCP clients.

LocalFree

Serena

31k

oraios/serena

Coding agent that uses language servers for symbolic code operations.

LocalFreePython

Mastra/

29k

mastra-ai/mastra

Client implementation for Mastra, providing seamless integration with MCP-compatible AI models and tools.

LocalFree

Desktop Commander

10k

wonderwhy-er/DesktopCommanderMCP

Manage and execute programs, and read, write, search and edit code and text files.

LocalFreeTypeScript

Ouroboros

6.3k

Q00/ouroboros

Local agent OS for spec-first AI coding: a replayable, ledger-recorded interview, seed, execute, evaluate and evolve loop that MCP clients can drive.

LocalFreePython

Lingo.dev

5.4k

lingodotdev/lingo.dev

Make your AI agent speak every language on the planet, using Lingo.dev Localization Engine.

LocalFree