Eval Model

Measures the task accuracy of text models served by MAX using standard benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and BABILong. Use when benchmarking a served model, comparing it with model-card or reference scores, verifying that a new MAX model produces correct answers, or running repeatable dataset evaluations against a MAX OpenAI-compatible endpoint.

213 stars 22 forksAI, LLMs & Agents Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add modular/skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add modular/skills --skill eval-model
Manual install
git clone --depth 1 https://github.com/modular/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/eval-model ~/.claude/skills/eval-model

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Evaluate a model's accuracy on MAX

A model that serves cleanly can still answer benchmark questions wrong. This
skill measures how accurately a text model behind MAX answers standard
datasets — GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and
BABILong. It checks endpoint compatibility before each run, separates serving
failures from wrong answers, and writes reproducible per-task scores you can
compare against a model card.

**Use this skill when** you're benchmarking a served model, comparing it with
model-card or reference scores, verifying that a newly imported MAX model answers
correctly, or running repeatable dataset evaluations against a MAX
OpenAI-compatible endpoint.

**Do not use this skill when** the model doesn't serve yet. Bring the model up
first (`import-model`); if it serves but generates wrong output, chase the
divergence with `debug-model` before benchmarking.

## References

| File | Read when |
|------|-----------|
| [references/examples.md](references/examples.md) | You want more command variants — remote endpoints, artifact paths, long context |
| [references/fast-fail-thresholds.md](references/fast-fail-thresholds.md) | Before changing seed floors or adding a seed task |
| [references/heartbeat-protocol.md](references/heartbeat-protocol.md) | Building a program that consumes the progress stream |

Read the reference for what you're doing, not all of them upfront.

## Set up the evaluator
Read the full skill on GitHub

Details

Skill name
eval-model
Repository
modular/skills
Path
eval-model/SKILL.md
Repo updated
18 November 2026

FAQ

What does the Eval Model skill do?

Measures the task accuracy of text models served by MAX using standard benchmarks such as GSM8K, MMLU, HellaSwag, ARC, AIME, GPQA, TruthfulQA, WinoGrande, and BABILong. Use when benchmarking a served model, comparing it with model-card or reference scores, verifying that a new MAX model produces correct answers, or running repeatable dataset evaluations against a MAX OpenAI-compatible endpoint.

How do I install the Eval Model skill?

Claude Code plugin: /plugin marketplace add modular/skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Eval Model popular?

It lives in modular/skills, which has 213 GitHub stars and 22 forks.

Related skills

Benchmark Model

213

modular/skills

Benchmark a model served on MAX with the `max benchmark` command: measure throughput (tokens/sec), latency (TTFT, TPOT, inter-token latency), and GPU utilization by driving load against a running `max serve` endpoint. Use this whenever the user wants to benchmark, load-test, or measure the performance of a MAX model, get tokens-per-second / TTFT / TPOT numbers, run a concurrency or request-rate sweep, compare latency vs throughput, size a deployment, or produce benchmark JSON, even if they don't say "benchmark" by name. Also use when a `max benchmark` run fails to connect or reports zero/garba

AI, LLMs & Agents22

Debug Model

213

modular/skills

Debug a MAX model that loads, compiles, serves, and generates tokens but whose output disagrees with a reference implementation. Use whenever parity debugging stalls on scalar taps, the model returns gibberish or wrong greedy tokens, logit cosine is high but argmax differs, or generation is coherent then diverges. Applies during an architecture port, a quantization bring-up, a multi-GPU conversion, or after a MAX upgrade. Triggers on "parity failure", "silent corruption", "logits match but tokens diverge", "top-1 mismatch", "greedy divergence", and "model serves but generates garbage". Not for

AI, LLMs & Agents22

Import Model

213

modular/skills

Use when importing a new model architecture into MAX from a Hugging Face model ID. Triggers on: "import a model into MAX", "add model to MAX", "bring up <HF model> in MAX". Workflow: inspect the Hugging Face config and modeling code, then scaffold from a similar ModuleV3 architecture. Implement each layer as a max.nn Module that matches HF, serve, then verify against the Hugging Face reference. When the server runs but output is wrong (gibberish, greedy mismatch, coherent-then-diverges), load debug-model for the divergence hunt.

AI, LLMs & Agents22

Excalidraw MCP Diagramming

206

thomast1906/github-copilot-agent-skills

Create and edit diagrams on a live Excalidraw canvas using the Excalidraw MCP server. Use when asked to draw, diagram, sketch, or visualise architectures, workflows, data flows, system designs, flowcharts, mind maps, or sequence diagrams. Trigger phrases include "create an excalidraw", "draw me a diagram", "make a flowchart", "visualise the system", "diagram this architecture", "export to PNG/SVG". Can export to PNG, SVG, .excalidraw file, or a shareable URL. Do NOT use for Draw.io or diagrams.net output (use drawio-mcp-diagramming instead).

AI, LLMs & Agents54

Drawio MCP Diagramming

206

thomast1906/github-copilot-agent-skills

Create and edit diagrams using the Draw.io MCP server — any shape, any vendor. USE FOR: draw me a diagram, create an architecture diagram, add Azure/AWS/GCP/Cisco/Kubernetes icons to a diagram, convert Mermaid to draw.io, fix overlapping arrows, edit a .drawio file, network topology diagrams, CI/CD pipeline diagrams, auth flow diagrams. Supports XML, Mermaid, and CSV. Uses drawio/search_shapes to find any of 10,000+ shapes across all vendor and icon libraries. DO NOT USE FOR: Excalidraw output (use excalidraw-mcp-diagramming skill).

AI, LLMs & Agents54

Migrate Max V2 to V3

213

modular/skills

Use when migrating a MAX model from the V2 graph API (max.nn.legacy, TensorValue, explicit Graph construction) to ModuleV3 (max.nn, Tensor, F.lazy() + compile()). Triggers on: "migrate to ModuleV3", "port a max.nn.legacy model to max.experimental", "convert TensorValue to Tensor", "update my model to the V3 API", "ModuleV3 migration". Workflow: read the concept map, rewrite imports and __call__ signatures, and replace graph building with compile(weights=...). Align weight names with the new module hierarchy and move the KV cache unflatten into forward(). Port sharding to a DeviceMesh when the

AI, LLMs & Agents22

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.