Benchmark Honestly

Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance"

517 stars 39 forksSoftware Development

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add dotslashsimran/simcraft --skill benchmark-honestly
Manual install
git clone --depth 1 https://github.com/dotslashsimran/simcraft.git /tmp/simcraft
mkdir -p ~/.claude/skills
cp -r /tmp/simcraft/skills/coding/benchmark-honestly ~/.claude/skills/benchmark-honestly

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Benchmark honestly

A benchmark is a claim: "under these conditions, A takes this long, and B is this much faster, give or take this much". Most published benchmark numbers are wrong because one of those parts is missing. This skill makes every part explicit and runs checks that catch the usual mistakes. Deeper explanations are in `reference.md`.

Scripts live in `~/simcraft/skills/coding/benchmark-honestly/scripts/`. They need only Python 3.9+.

## 1. Write the question down first

Before running anything, agree on these with the user and write them at the top of the report:
- **What is measured:** throughput (work per second) or latency (time per request). For latency, **which percentile** matters (p50 for typical, p99 or p99.9 for the tail users complain about).
- **Cold or warm:** first run after start (caches empty, JIT not compiled) or steady state. These are different questions with different answers.
- **The workload:** realistic input sizes and shapes. A 100-element input fits in L1 cache and says nothing about a 10 GB one.
- **What difference matters:** "a 2% gain is worth shipping" or "only 20% matters". Pass it as `--min-effect`.
- **Microbenchmark or end to end:** a 3x faster function inside a request that spends 95% of its time on the network changes nothing. Measure end to end before celebrating a micro result.

## 2. Pick the tool

Use the ecosystem's own tool if it is installed, because it runs in-process with less overhead:
Read the full skill on GitHub

Details

Skill name
benchmark-honestly
Path
skills/coding/benchmark-honestly/SKILL.md
License
MIT
Repo updated
9 November 2026

FAQ

What does the Benchmark Honestly skill do?

Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance"

How do I install the Benchmark Honestly skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add dotslashsimran/simcraft --skill benchmark-honestly. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Benchmark Honestly popular?

It lives in dotslashsimran/simcraft, which has 517 GitHub stars and 39 forks.

Related skills

API Design Review

517

dotslashsimran/simcraft

Review the design of an API surface, meaning an HTTP API (OpenAPI spec or route code), an SDK's public classes and functions, or a CLI. Checks naming and consistency, resource modelling, error design (stable codes, RFC 9457 problem details, request ids), versioning and breaking changes, pagination, filtering, idempotency and retries, rate limits, timeouts, long-running operations, webhooks, auth, partial updates, ID, time and money formats, SDK ergonomics and CLI conventions. Uses scripts to lint OpenAPI files, classify changes between two spec versions as breaking or safe, and dump an SDK's p

Software Development39

Bisect Anything

517

dotslashsimran/simcraft

Find the exact commit (or dependency version, config value, or input) that caused a regression by turning it into an automatic pass/fail check and running git bisect run in a separate worktree, then shrink the bad commit to the hunk that matters and explain it. Handles failing tests, changed output, slowdowns, crashes, visual changes, server behaviour, flaky checks, unbuildable commits and merge-heavy history. Use when the user says "what broke this", "find the commit that", "when did this start", "it used to work", "regression", "git bisect", "which version broke it", or "which change made it

Git & Code Review39

Brand Kit

517

dotslashsimran/simcraft

Learn a company's brand design from its live website (colours, fonts with the real font files, logo, button and card shapes, texture, image style) and save it as a reusable kit that vox-explainer and image-gen apply automatically. Use FIRST, before any brand or sponsored video, sticker, graphic or asset for a company (e.g. "a video for Devin", "ChatGPT-style stickers", "match their brand"), or when asked to get, learn, extract or match a brand's look.

Frontend & Design39

Code Loop

517

dotslashsimran/simcraft

Make a seamless, perfectly looping generative video in code, from a three.js scene or a raw GLSL fragment shader, rendered frame-perfect with jittered anti-aliasing and motion blur, then checked for a visible seam, freezing and blinking. Use when asked for a "seamless loop", a "perfect loop", "generative art video", a "shader loop", "three.js animation to video", to "render a looping animation", a looping GIF from code, or any code-driven animation that has to loop without a visible cut.

Software Development39

Codebase Map

517

dotslashsimran/simcraft

Understand an unfamiliar codebase fast, the way senior engineers do. Reads the orientation signals (README, manifests, CI, Dockerfiles, routes, migrations), mines git history for hotspots, hidden coupling and ownership, builds the import graph to find layers, cycles, load-bearing and dead code, then traces one real request or command end to end by actually running it. Writes MAP.md with a one-paragraph purpose, a Mermaid architecture diagram, the traced path as file:line hops, hotspot and coupling tables, a glossary, gotchas and first tasks, plus an optional SVG. Use when the user says "explai

Software Development39

Tree Generator

360

TanStack/intent

Use when turning an approved domain_map.yaml and skill_spec.md into a full-library skill tree or explicitly auditing its structure and dependencies. For agreed batches and source-driven guidance updates, use generate-skill instead.

Software Development26

Browse the most popular Claude skills, all software development skills, or write your own with the skills generator.