LLM Eval Harness MCP Server: Install & Setup

Install LLM Eval Harness in Claude Code

Claude Code

claude mcp add llm-eval-harness -- docker run -i --rm ghcr.io/razilzul/llm-eval-harness:1.0.0

mcp.json (Claude Desktop, Cursor, VS Code, Windsurf)

{
  "mcpServers": {
    "llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ]
    }
  }
}

Frequently asked questions

What does the LLM Eval Harness MCP server do?

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Is the LLM Eval Harness MCP server free?

Yes, most likely. Open source and no paid account or API key is mentioned in the listing. Check the provider for current pricing.

Does LLM Eval Harness run locally or remotely?

Local. Runs on your machine (stdio). You install it with npx, uvx, Docker or from source.

How do I add LLM Eval Harness to Claude Code?

Run: claude mcp add llm-eval-harness -- docker run -i --rm ghcr.io/razilzul/llm-eval-harness:1.0.0

More ai agents & coding agents MCP servers

View all

llm-oracle

weiseer/llm-oracle

Up-to-date LLM pricing + availability catalog with cited sources for AI agent routing.

LocalPaid service

Living Stack Community

equinoxaifinance-rgb/living-stack-community

Save your coding agent's exact next step and load it after a restart. Free MCP server, no account.

LocalFree

llmshift

DeChuks-hub/llmshift

Find retiring LLM calls in code and test the replacement model on real traffic before switching.

LocalFree

Literature Review Council

Ravicha2/lit-review-council

Multi-agent literature review: research, peer review, and synthesize grounded reports.

LocalAccount needed

Localecheck

0130

qinisolabs/localecheck

Verified locale data for AI agents: dates, phone, currency, tax, holidays, addresses (UK/US)

LocalFree

LetsClarify

heisee/LetsClarify

Human-in-the-Loop API: AI agents create forms, share URLs with humans, get JSON back.

LocalAccount needed