Omni AI Eval

Evaluate Omni AI accuracy using Omni's built-in eval system — define a prompt set, run a judged eval against a model (or branch), and read the accuracy-judge verdicts. Use this skill whenever someone wants to evaluate Omni AI, benchmark Blobby, run regression tests, compare AI output across branches or model-context changes, measure AI quality, run A/B tests on model changes, assess the impact of an ai_context or modeling change, or any variant of "run evals", "test Blobby", "benchmark query generation", "compare AI results", "regression test", "how accurate is the AI", or "measure the impact

46 stars 10 forksAI, LLMs & Agents Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add exploreomni/omni-agent-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add exploreomni/omni-agent-skills --skill omni-ai-eval
Manual install
git clone --depth 1 https://github.com/exploreomni/omni-agent-skills.git /tmp/omni-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/omni-agent-skills/skills/omni-ai-eval ~/.claude/skills/omni-ai-eval

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Omni AI Eval

Omni ships a first-class eval system (the **AI Hub** → **Prompt sets** and **Eval runs**). This skill drives it through the Omni CLI: define a reusable **prompt set**, start a judged **eval run** against a model or branch, and read per-prompt verdicts from Omni's built-in **accuracy judge**.

Prefer this native system over building your own harness. The judge scores each answer *semantically* against the full agent conversation — it does not require golden query JSON, and it evaluates the whole agentic workflow (topic selection, queries, results, and the final written answer), not just generated query structure.

> **Tip**: Use `omni-ai-optimizer` to improve scores after finding failures, `omni-model-builder` to apply context changes on a branch before A/B testing, and `omni-model-explorer` to discover topics and fields when writing prompts.

## Prerequisites

```bash
# Verify the Omni CLI is installed — if not, ask the user to install it.
# See: https://github.com/exploreomni/cli#readme
command -v omni >/dev/null || echo "ERROR: Omni CLI is not installed."

# Verify the CLI has the eval commands. If this errors with "unknown command",
# the binary is stale — ask the user to update it (the ai-eval group is generated
# from the bundled API spec).
omni ai-eval --help >/dev/null 2>&1 || echo "ERROR: 'omni ai-eval' missing — update the Omni CLI."
```
Read the full skill on GitHub

Details

Skill name
omni-ai-eval
Path
skills/omni-ai-eval/SKILL.md
Repo updated
16 November 2026

FAQ

What does the Omni AI Eval skill do?

Evaluate Omni AI accuracy using Omni's built-in eval system — define a prompt set, run a judged eval against a model (or branch), and read the accuracy-judge verdicts. Use this skill whenever someone wants to evaluate Omni AI, benchmark Blobby, run regression tests, compare AI output across branches or model-context changes, measure AI quality, run A/B tests on model changes, assess the impact of an ai_context or modeling change, or any variant of "run evals", "test Blobby", "benchmark query generation", "compare AI results", "regression test", "how accurate is the AI", or "measure the impact

How do I install the Omni AI Eval skill?

Claude Code plugin: /plugin marketplace add exploreomni/omni-agent-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Omni AI Eval popular?

It lives in exploreomni/omni-agent-skills, which has 46 GitHub stars and 10 forks.

Related skills

Omni Embed

46

exploreomni/omni-agent-skills

Embed Omni Analytics dashboards in external applications — URL signing, custom themes, iframe events, entity workspaces, and permission-aware content — using the @omni-co/embed SDK and Omni CLI. Use this skill whenever someone wants to embed a dashboard, sign an embed URL, customize the embedded theme, handle embed events, listen for clicks or drills in the iframe, send filters to an embedded dashboard, set up entity workspaces, look up embed users, build a permission-aware content list, white-label an embedded dashboard, or any variant of "embed this dashboard", "customize the iframe theme",

Frontend & Design10

Omni Admin

46

exploreomni/omni-agent-skills

Administer an Omni Analytics instance — manage connections, users, groups, user attributes, permissions, schedules, and schema refreshes via the Omni CLI. Use this skill whenever someone wants to manage users or groups, set up permissions on a dashboard or folder, configure user attributes, create or modify schedules, manage database connections, refresh a schema, set up access controls, provision users, or any variant of "add a user", "give access to", "set up permissions", "who has access", "configure connection", "refresh the schema", or "schedule a delivery".

Data & Analytics10

Omni AI Optimizer

46

exploreomni/omni-agent-skills

Optimize an Omni Analytics model for Blobby, the Omni Agent — configure ai_context, ai_fields, synonyms, sample_queries, and ai_settings. Use this skill whenever someone wants to improve AI accuracy in Omni, make Blobby smarter, add AI context or example questions, curate which fields the AI sees, personalize AI context by user attribute, diagnose context-window pruning or truncation, raise a topic's AI context budget, control which topics AI can reach, troubleshoot why Blobby gives wrong answers, or any variant of "make the AI better", "Blobby isn't answering correctly", "optimize for AI", or

AI, LLMs & Agents10

H3 Prompt Writing

44

fengyungithub/dsh-short-video-studio

Write MiniMax H3 video generation prompts for T2VA, I2VA, FL2VA, L2VA, and Ref2VA. Use when rewriting multimodal requests into H3 prompt structures, composing integrated_multimodal_description, overall_soundscape, and non_diegetic_music, aligning keyframes, or defining reference labels for images, videos, and audio. dsh-short-video-studio 适配版:还负责把画布上的镜头表/分镜/参考绑定重写为 H3 结构化 prompt(见文末「插件对接」适配节)。

AI, LLMs & Agents

Omni Model Builder

46

exploreomni/omni-agent-skills

Create and edit Omni Analytics semantic model definitions — views, topics, dimensions, measures, relationships, query views, composite topics, and aggregate tables (materialized_query) — using YAML through the Omni CLI. Use this skill whenever someone wants to add a field, create a new dimension or measure, define a topic, set up joins between tables, modify the data model, build a new view, add a calculated field, create a relationship, edit YAML, work on a branch, promote model changes, or any variant of "model this data", "add this metric", "create a view for", or "set up a join between". A

AI, LLMs & Agents10

Claude Agent SDK

35

liangdabiao/claudesdk-skill

Claude Agent SDK framework guide for building AI-powered applications. Provides comprehensive API knowledge, architecture patterns, and code examples for both TypeScript and Python. Use when: (1) building projects with @anthropic-ai/claude-agent-sdk, (2) creating AI agents with tool use, hooks, subagents, or MCP servers, (3) implementing multi-turn conversations, streaming responses, or session management, (4) configuring permission modes, custom tools, or agent orchestration, (5) debugging or extending existing Claude Agent SDK code, (6) user mentions "agent sdk", "claude agent", "claude-agen

AI, LLMs & Agents

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.