Search Benchmark

Generate a search quality benchmark for the AI Registry. Generates ground truth from the registry's assets, runs 100+ queries against the semantic search API, evaluates results using NDCG@10/MRR/Recall, and produces a markdown report. Use when you want to measure search quality after changes to the scoring algorithm, embedding model, or indexed content.

1,052 stars 263 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark
Manual install
git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git /tmp/mcp-gateway-registry
mkdir -p ~/.claude/skills
cp -r /tmp/mcp-gateway-registry/.claude/skills/search-benchmark ~/.claude/skills/search-benchmark

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Search Benchmark Skill

Measure semantic search quality against a deployed AI Registry. Generates a ground truth dataset from the registry's own assets, runs queries against the live API, evaluates results using standard information retrieval metrics (NDCG@10, MRR, Recall@10), and produces a markdown report.

## Prerequisites

1. **Registry URL** - The base URL of the deployed registry (e.g., `https://d2xl2zfuhgc4l0.cloudfront.net`)
2. **JWT Token** - A valid admin token in `.token` file (get from "Get JWT Token" button in registry UI)
3. **Registry must have assets indexed** - At least some servers, agents, or skills registered

The `.token` file supports both raw JWT format and the full JSON response from the registry UI.

## Input

```
/search-benchmark [REGISTRY_URL] [TOKEN_FILE]
```

- **REGISTRY_URL** - Base URL of the registry to benchmark (default: reads from user or uses `http://localhost`)
- **TOKEN_FILE** - Path to the token file (default: `.token`)

## Workflow

### Step 1: Check for Existing Ground Truth

Check if a ground truth dataset already exists:

```bash
ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/null
```

If the file exists, report how many queries it contains and ask the user: "A ground truth dataset already exists (N queries). Do you want to use it or generate a new one from this registry?"

- If **use existing**: skip to Step 2
- If **generate new**: proceed to Step 1b

### Step 1b: Generate Expert Ground Truth
Read the full skill on GitHub

Details

Skill name
search-benchmark
Path
.claude/skills/search-benchmark/SKILL.md
License
Apache-2.0
Repo updated
18 November 2026

FAQ

What does the Search Benchmark skill do?

Generate a search quality benchmark for the AI Registry. Generates ground truth from the registry's assets, runs 100+ queries against the semantic search API, evaluates results using NDCG@10/MRR/Recall, and produces a markdown report. Use when you want to measure search quality after changes to the scoring algorithm, embedding model, or indexed content.

How do I install the Search Benchmark skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Search Benchmark popular?

It lives in agentic-community/mcp-gateway-registry, which has 1,052 GitHub stars and 263 forks.

Related skills

Agentcore Register

1.1k

agentic-community/mcp-gateway-registry

Given an MCP server URL, probe the server via curl to discover its metadata and tools, then generate a markdown file with copy-pasteable content for each field in the Amazon Bedrock AgentCore "Create record" form.

AI, LLMs & Agents263

Benchmark Report

1.1k

agentic-community/mcp-gateway-registry

Generate a benchmark report from stress test results (registration, API performance, search concurrency). Reads JSON result files and produces a markdown report suitable for docs/benchmarks/.

Testing & QA263

Create Milestone

1.1k

agentic-community/mcp-gateway-registry

Create a GitHub milestone for an upcoming release. Suggests the next version based on the latest release, gathers all merged PRs and closed issues since that release, presents a draft with two tables (Issues and PRs) for user approval, then creates the milestone and assigns all approved items.

Git & Code Review263

Search Registry

1.1k

agentic-community/mcp-gateway-registry

Search the MCP Gateway Registry using semantic search. Takes a natural language query, calls the /api/search/semantic endpoint, and returns formatted results directly in the conversation.

AI, LLMs & Agents263

Macos Setup

1.1k

agentic-community/mcp-gateway-registry

Complete macOS setup and teardown for MCP Gateway & Registry (AI-registry). Clones the repository, installs all services, configures Keycloak auth, registers the Cloudflare docs server, and verifies the full stack. Also supports complete teardown. Can be run directly from its GitHub URL without the repository already cloned. Uses an interactive or default-values mode chosen at startup.

AI, LLMs & Agents263

Code Review AI AI Review

2k

rmyndharis/antigravity-skills

Combine static analysis with model-assisted code review to assess security, performance, correctness, and architecture. Use when designing review automation or reviewing complex changes.

AI, LLMs & Agents295

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.