Calibrating Thresholds and Baselines

Check that a score-threshold-flag pipeline actually separates anything before you trust or tune it — reference constants that are hand-authored but formatted as measurements, a computed feature whose units do not match the constant it is compared against, per-class fire rates that expose a flag firing on nearly every item of both classes, rate features confounded by item length, confidence scores where one feature family contributes once per sub-feature, threshold sweeps hardcoded to one comparison direction and one statistic, and separating features that carry the decision from features that

115 stars 18 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add wdm0006/python-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add wdm0006/python-skills --skill calibrating-thresholds-and-baselines
Manual install
git clone --depth 1 https://github.com/wdm0006/python-skills.git /tmp/python-skills
mkdir -p ~/.claude/skills
cp -r /tmp/python-skills/skills/common/threshold-calibration ~/.claude/skills/calibrating-thresholds-and-baselines

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Calibrating Thresholds and Baselines

A scoring pipeline has three layers: features computed per item, reference
constants the features are compared against, and a flag/score layer that turns
the comparison into something a user reads. **reporting-derived-metrics** covers
the first layer — what a feature function returns when the sample is too thin to
support it. This skill covers the other two: where the constant came from, and
whether the flag it drives carries any information at all.

The failure mode is not a crash. It is a pipeline that runs clean, emits
confident-looking flags on every input, and nobody notices because no one ever
asked how often each flag fires per class.

## A committed baseline file is not a measurement

Baselines get checked in as JSON or a module constant and immediately start
reading as ground truth, because they have the shape of data:

```json
{
  "name": "general_prose",
  "created": "2023-09-04",
  "sample_size": 750,
  "features": {
    "avg_item_len": {"mean": 12.4, "sd": 5.1},
    "diversity":    {"mean": 0.61, "sd": 0.07},
    "rate_per_word":{"mean": 0.22, "sd": 0.05}
  }
}
```

Every number there is two significant figures, `sample_size` is a round 750, and
nothing in the repository produces the file. It was typed. Downstream, `z =
(value - 0.22) / 0.05` is reported to four decimals and quoted in an issue as
evidence.

Before trusting a stored baseline, ask four questions:
Read the full skill on GitHub

Details

Skill name
calibrating-thresholds-and-baselines
Path
skills/common/threshold-calibration/SKILL.md
License
MIT
Repo updated
18 November 2026

FAQ

What does the Calibrating Thresholds and Baselines skill do?

Check that a score-threshold-flag pipeline actually separates anything before you trust or tune it — reference constants that are hand-authored but formatted as measurements, a computed feature whose units do not match the constant it is compared against, per-class fire rates that expose a flag firing on nearly every item of both classes, rate features confounded by item length, confidence scores where one feature family contributes once per sub-feature, threshold sweeps hardcoded to one comparison direction and one statistic, and separating features that carry the decision from features that

How do I install the Calibrating Thresholds and Baselines skill?

Claude Code plugin: /plugin marketplace add wdm0006/python-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Calibrating Thresholds and Baselines popular?

It lives in wdm0006/python-skills, which has 115 GitHub stars and 18 forks.

Related skills

Auditing Python Security

115

wdm0006/python-skills

Audits Python libraries for security vulnerabilities using Bandit, pip-audit, Semgrep, and detect-secrets. Identifies SQL injection, command injection, hardcoded credentials, secrets exposed through tracebacks, weak cryptography, and insecure deserialization. Use when reviewing library security, setting up security scanning in CI, or implementing secure coding patterns.

Security18

Building Accessible Interfaces

115

wdm0006/python-skills

Makes web and app UIs usable by keyboard, screen-reader, low-vision and motion-sensitive users — semantic HTML before ARIA, labelled controls, visible focus, WCAG AA contrast measured rather than eyeballed, touch targets, reduced motion, and automated axe checks plus a manual keyboard pass in CI or before every UI PR. Use when building or restyling a screen, adding a form, modal, menu or custom control, choosing colors, or reviewing a UI change.

Frontend & Design18

Building Browser Extensions

115

wdm0006/python-skills

Builds and tests Manifest V3 browser extensions — service-worker message channels that survive teardown, chrome.storage read-modify-write safety, settings validation in the popup, and a Jest chrome mock that can actually fail. Use when writing or reviewing a background service worker, a popup that reads/writes chrome.storage, an extension's Jest suite, or debugging an extension whose UI silently stops responding.

Software Development18

Reporting Derived Metrics

115

wdm0006/python-skills

Compute statistics, scores, and flags from samples that may be too small to support them — undefined dispersion returned as 0.0 and tripping a minimum threshold, sentinel choice (None vs 0 vs NaN), threshold blocks gated on "was this measured", reports that narrate findings from absent data, nullability as a public API change, `is None` vs truthiness, broad excepts that turn a metric bug into a normal-shaped result, and heavy-tailed samples where the mean points the opposite way to the median. Use when writing or reviewing a scoring/analysis pipeline, a z-score or outlier check, a quality or a

Data & Analytics18

Ielts Dashboard

93

xuchi-0808/ielts-claude-skills

IELTS 学习数据可视化 Dashboard。生成本地 HTML 网页,展示写作趋势图、四科雷达图、错题热力图、同义替换统计、考试倒计时。 触发方式:/ielts-dashboard、「Dashboard」「看数据」「打开数据面板」

Data & Analytics

Cleanup Dashboards

81

TomGranot/hubspot-admin-skills

Audit and consolidate HubSpot reporting dashboards. Identifies unused, duplicate, or outdated dashboards. Must be performed manually — no dashboard API is available.

Data & AnalyticsIn 1 list19

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.