Information Theory

Use this skill when the user says 'entropy', 'cross-entropy loss', 'KL divergence', 'perplexity', 'bits per byte', 'mutual information', 'compression is prediction', 'manifold hypothesis', 'intrinsic dimension', 'ELBO'. Explains the information-theoretic quantities behind ML training and evaluation and applies them: interpreting losses and perplexity, KL in distillation/RLHF/VAEs, calibration, and measuring intrinsic dimension of embeddings. Do NOT use for: diffusion/score-based generative math (fundamental-truths/statistical-mechanics), classical statistics and hypothesis tests (data-science/

45 stars 6 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill information-theory
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/fundamental-truths/information-theory ~/.claude/skills/information-theory

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Information Theory for ML

## Purpose
Language-model training minimizes cross-entropy, distillation and RLHF use KL divergence, VAEs maximize an ELBO, and perplexity and bits-per-byte are information measures. Knowing exactly what these quantities mean prevents common errors: comparing perplexities across tokenizers, mixing nats and bits, using KL in the wrong direction, or over-reading the manifold hypothesis.

## Agent Protocol

### Trigger
"entropy", "cross-entropy", "KL divergence", "forward vs reverse KL", "perplexity", "bits per byte", "mutual information", "ELBO", "manifold hypothesis", "intrinsic dimension".

### Input Context
- The model/loss in question and its units (nats vs bits).
- Tokenizer details when comparing language models.
- Data samples or embeddings if measuring.

### Output Artifact
Explanation with the exact formula, numeric interpretation, and code to compute the quantity correctly.

### Response Format
```
Quantity: {name} = {formula}  Units: {nats|bits}
Interpretation: {what the number means here}
Pitfall: {common mistake}
Compute: {code snippet / library call}
```

### Completion Criteria
- [ ] Units stated and conversions explicit (1 nat = 1/ln 2 ≈ 1.4427 bits).
- [ ] Direction of KL stated and justified.
- [ ] Cross-model comparisons normalized (bits per byte/character) when tokenizers differ.

## Core Quantities
| Quantity | Definition | Notes |
|---|---|---|
Read the full skill on GitHub

Details

Skill name
information-theory
Path
skills/fundamental-truths/information-theory/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Information Theory skill do?

Use this skill when the user says 'entropy', 'cross-entropy loss', 'KL divergence', 'perplexity', 'bits per byte', 'mutual information', 'compression is prediction', 'manifold hypothesis', 'intrinsic dimension', 'ELBO'. Explains the information-theoretic quantities behind ML training and evaluation and applies them: interpreting losses and perplexity, KL in distillation/RLHF/VAEs, calibration, and measuring intrinsic dimension of embeddings. Do NOT use for: diffusion/score-based generative math (fundamental-truths/statistical-mechanics), classical statistics and hypothesis tests (data-science/

How do I install the Information Theory skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill information-theory. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Information Theory popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Statistical Mechanics

45

j4flmao/agent-skills

Use this skill when the user says 'diffusion model math', 'score matching', 'DDPM', 'Langevin dynamics', 'reverse SDE', 'probability flow ODE', 'Boltzmann distribution', 'energy-based model', 'classifier-free guidance', 'flow matching', 'noise schedule'. Explains the statistical-mechanics view of generative modeling (Boltzmann distributions, Langevin dynamics, forward/reverse diffusion SDEs, score functions) and implements a correct DDPM training and sampling loop. Do NOT use for: using pretrained image generators (ai/genai-vision), information-theoretic losses (fundamental-truths/information-

Data & Analytics

Enterprise Data Governance

45

j4flmao/agent-skills

Use when the user asks about data governance, data classification, data ownership and stewardship, retention policy, data catalog policy or PII handling rules. Produces a governance framework with classification, owners and controls. Not for building pipelines (use data-etl-pipeline) or lineage tooling (use data-lineage).

Data & Analytics

Tableau Next Implementation Advisor

24

forcedotcom/partner-ecosystem-agentic-kit

Use for Tableau Next scratch enablement, Data 360 dependencies, semantic models, workspaces, visualizations, dashboards, deployment ordering, and integration decisions. Owns Tableau Next implementation and integration decisions relevant to partner solutions.

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.