AI Eval CI
Run AI agent and LLM evaluations in CI/CD pipelines — automated quality gates that fail the build when AI output quality drops. Use when someone asks to "test my AI agent", "add evals to CI", "catch prompt regressions", "compare models", "evaluate LLM output quality", "set up AI quality gates", or "benchmark my agent before deploying". Covers eval frameworks (Cobalt, Promptfoo, Braintrust), LLM-as-judge scoring, threshold-based assertions, and GitHub Actions integration.
Install
npx skills add TerminalSkills/skills --skill ai-eval-cigit clone --depth 1 https://github.com/TerminalSkills/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/skills/ai-eval-ci ~/.claude/skills/ai-eval-ciPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# AI Eval in CI ## Overview Test AI agents and LLM outputs the same way you test code — automated evaluations that run in CI, compare against baselines, and fail the build when quality drops. No dashboards to check manually. Just `npx eval run --ci` and a red or green build. ## When to Use - Adding quality gates before deploying AI features to production - Catching prompt regressions when system prompts or models change - Comparing model performance (GPT-4o vs Claude Sonnet vs local Llama) - Validating RAG pipeline accuracy against a test dataset - Benchmarking agent tool-calling accuracy and latency ## Instructions ### Strategy 1: Promptfoo (Config-Driven Evals) Promptfoo is the most popular open-source eval framework. Define test cases in YAML, run against multiple providers, get a comparison matrix.
Details
- Skill name
ai-eval-ci- Repository
- TerminalSkills/skills
- Path
skills/ai-eval-ci/SKILL.md- License
- Apache-2.0
- Repo updated
- 8 November 2026
FAQ
What does the AI Eval CI skill do?
Run AI agent and LLM evaluations in CI/CD pipelines — automated quality gates that fail the build when AI output quality drops. Use when someone asks to "test my AI agent", "add evals to CI", "catch prompt regressions", "compare models", "evaluate LLM output quality", "set up AI quality gates", or "benchmark my agent before deploying". Covers eval frameworks (Cobalt, Promptfoo, Braintrust), LLM-as-judge scoring, threshold-based assertions, and GitHub Actions integration.
How do I install the AI Eval CI skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add TerminalSkills/skills --skill ai-eval-ci. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is AI Eval CI popular?
It lives in TerminalSkills/skills, which has 213 GitHub stars and 34 forks.
Related skills
3dsmax Rendering
213TerminalSkills/skills
Scripts and configures production rendering in Autodesk 3ds Max with the V-Ray and Corona renderers: output size and files, render elements, denoising, light mix, batch and command-line rendering, and network rendering. Use when a user asks to set up a production render, render several cameras in one batch, render from the command line or on a render farm, add render passes for compositing, or cut render time for archviz and product shots.
3dsmax Scripting
213TerminalSkills/skills
Covers scripting Autodesk 3ds Max, the 3D modeling and rendering application, with MAXScript and Python (pymxs): scene manipulation, object creation, material assignment, camera and light setup, batch operations, and file I/O. Use when tasks involve automating repetitive 3ds Max workflows, batch processing scenes, running scripts headless with 3dsmaxbatch, creating custom tools, or scripting scene setup for archviz, product visualization, or VFX.
3proxy
213TerminalSkills/skills
3proxy is a small open-source proxy server that runs HTTP/HTTPS, SOCKS4/5, SNI and TCP/UDP port-mapping proxies from one config file. Use when a user asks to set up an HTTP or SOCKS5 proxy, add proxy users and passwords, write 3proxy access rules, chain or rotate upstream (parent) proxies, limit bandwidth, connections or monthly traffic per user, run 3proxy in Docker, or fix a 3proxy.cfg that will not start.
AI Guardrails
213TerminalSkills/skills
Adds safety guardrails to LLM applications in Python: prompt injection checks, a canary token for system-prompt leaks, OpenAI moderation, PII redaction, Pydantic output validation, rate and token limits, and grounding checks for RAG answers. Use when adding safety layers to a chatbot, detecting prompt injection, filtering harmful content, validating LLM output against a schema, or building a moderation pipeline.
Agentscope
213TerminalSkills/skills
Build and trace AI agents in Python with AgentScope, an open-source agent framework with a ReAct loop, tools, middleware, team pipelines and OpenTelemetry tracing. Use when building production agents that need observability, debugging an agent's model and tool calls, capping token spend per reply, or coordinating a leader with specialist agents.
AI Scientist
213TerminalSkills/skills
The AI Scientist-v2 is Sakana AI's open-source research agent that turns a topic description into research ideas, runs machine-learning experiments with an agentic tree search, and writes a LaTeX paper with citations. Use when someone wants to automate ML research, generate and test hypotheses, run "AI Scientist" or "AI-Scientist-v2", or study automated scientific discovery.
Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.