Agent Observability Spec

Specify the tracing, metrics, and alerting for an AI agent or LLM feature in production. Use when asked what to log for an LLM app, design agent tracing or spans, define quality and cost monitors, or answer 'how do we know if the agent is misbehaving?'. Produces an observability spec with a trace schema, metric definitions with owners and alert thresholds, sampling and retention policy, and a privacy note for logged content.

1,573 stars 281 forksAI, LLMs & Agents Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add mohitagw15856/pm-claude-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add mohitagw15856/pm-claude-skills --skill agent-observability-spec
Manual install
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git /tmp/pm-claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/pm-claude-skills/plugins/pm-agentops/skills/agent-observability-spec ~/.claude/skills/agent-observability-spec

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Agent Observability Spec Skill

You can't fix what you didn't record. For LLM systems the unit of observability is the *trace* — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.

> **Not quite this?** Use `agent-spec` when you are specifying what the agent should do before it is built.

## What This Skill Produces

- A **trace schema**: per-request spans and the fields each must carry
- **Metric definitions** across health, quality, cost, and behaviour — each with a threshold and owner
- A **sampling and retention policy** that keeps cost sane and debugging possible
- A **privacy note**: what logged content contains, who can see it, and how long it lives

## Required Inputs

Ask for (if not already provided):
- **The system's shape** — single LLM call, RAG pipeline, or multi-step tool-using agent
- **Traffic volume and cost sensitivity** — full tracing at 10M req/day is a budget decision
- **What "misbehaving" means here** — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
- **Existing observability stack** (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it

## Trace Schema

Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:

| Span | Must capture |
|---|---|
Read the full skill on GitHub

Details

Skill name
agent-observability-spec
Path
plugins/pm-agentops/skills/agent-observability-spec/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the Agent Observability Spec skill do?

Specify the tracing, metrics, and alerting for an AI agent or LLM feature in production. Use when asked what to log for an LLM app, design agent tracing or spans, define quality and cost monitors, or answer 'how do we know if the agent is misbehaving?'. Produces an observability spec with a trace schema, metric definitions with owners and alert thresholds, sampling and retention policy, and a privacy note for logged content.

How do I install the Agent Observability Spec skill?

Claude Code plugin: /plugin marketplace add mohitagw15856/pm-claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Agent Observability Spec popular?

It lives in mohitagw15856/pm-claude-skills, which has 1,573 GitHub stars and 281 forks, and is included in 1 curated skill list.

Related skills

Ab Test Planner

1.6k

mohitagw15856/pm-claude-skills

Diseña pruebas A/B estadísticamente rigurosas para características de producto, cambios de UI, flujos de onboarding y experimentos de precios. Úsalo cuando te pidan configurar un experimento, diseñar una prueba A/B, calcular tamaño de muestra o interpretar resultados de pruebas. Produce un plan de prueba completo con hipótesis, definición de variantes, tamaño de muestra, estimación de duración, métricas de seguridad y una guía de interpretación de resultados.

Testing & QAIn 1 list281

Ab Test Planner

1.6k

mohitagw15856/pm-claude-skills

प्रोडक्ट फ़ीचर, UI बदलाव, ऑनबोर्डिंग फ़्लो और प्राइसिंग प्रयोगों के लिए सांख्यिकीय रूप से कठोर A/B टेस्ट डिज़ाइन करें। जब कोई प्रयोग सेट करने, A/B टेस्ट डिज़ाइन करने, सैंपल साइज़ निकालने या टेस्ट के नतीजे समझाने को कहे, तब इसका उपयोग करें। परिकल्पना, वेरिएंट की परिभाषा, सैंपल साइज़, अवधि का अनुमान, गार्डरेल मेट्रिक और नतीजों की व्याख्या की गाइड के साथ एक पूरा टेस्ट प्लान तैयार करता है।

Testing & QAIn 1 list281

Accessible Travel Planner

1.6k

mohitagw15856/pm-claude-skills

Summarise what Accessible Travel Planner does in one line. Use when asked to [trigger phrases the user would say]. Produces [the concrete artifact].

Frontend & DesignIn 1 list281

Agent Project File

1.6k

mohitagw15856/pm-claude-skills

Set up the project file an AI agent keeps for work that runs over days or weeks: the goal, the definition of done, milestones, the current state, decisions made, open questions, and the exact point to resume from after an interruption. Use when asked to plan a long-running agent task, make my agent work on this over a week, keep state between agent sessions, or hand a multi-day project to an AI. Produces the project file, the update rules, the check-in schedule and the resume checklist.

AI, LLMs & AgentsIn 1 list281

Agent Negotiation Brief

1.6k

mohitagw15856/pm-claude-skills

Brief an AI agent that will negotiate for you with another party's agent or a person (a refund, a price, a meeting time, a contract term): your goal, your opening, your limits, your walk-away point, what it must never reveal, and when it must come back to you. Use when asked to let my agent negotiate, brief my AI for a negotiation, set limits for an agent bargaining on my behalf, or prepare for agent-to-agent negotiation. Produces the brief, the concession ladder, the never-reveal list and the hand-back triggers.

AI, LLMs & AgentsIn 1 list281

Agent Readiness Audit

1.6k

mohitagw15856/pm-claude-skills

Audit whether AI agents can actually use your product — docs, APIs, onboarding, errors, and discoverability, evaluated from a non-human user's perspective. Use when asked if a product is agent-ready, to audit a site or API for AI usability, to prepare for agentic traffic, or when agents keep failing against your product. Produces a scored readiness report with per-surface findings and a prioritised fix list. For optimising a single article for AI citation use aeo-optimizer; for designing the MCP server itself use mcp-server-spec.

AI, LLMs & AgentsIn 1 list281

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.