Benchmark Honestly
Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance"
Install
npx skills add dotslashsimran/simcraft --skill benchmark-honestlygit clone --depth 1 https://github.com/dotslashsimran/simcraft.git /tmp/simcraft
mkdir -p ~/.claude/skills
cp -r /tmp/simcraft/skills/coding/benchmark-honestly ~/.claude/skills/benchmark-honestlyPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Benchmark honestly A benchmark is a claim: "under these conditions, A takes this long, and B is this much faster, give or take this much". Most published benchmark numbers are wrong because one of those parts is missing. This skill makes every part explicit and runs checks that catch the usual mistakes. Deeper explanations are in `reference.md`. Scripts live in `~/simcraft/skills/coding/benchmark-honestly/scripts/`. They need only Python 3.9+. ## 1. Write the question down first Before running anything, agree on these with the user and write them at the top of the report: - **What is measured:** throughput (work per second) or latency (time per request). For latency, **which percentile** matters (p50 for typical, p99 or p99.9 for the tail users complain about). - **Cold or warm:** first run after start (caches empty, JIT not compiled) or steady state. These are different questions with different answers. - **The workload:** realistic input sizes and shapes. A 100-element input fits in L1 cache and says nothing about a 10 GB one. - **What difference matters:** "a 2% gain is worth shipping" or "only 20% matters". Pass it as `--min-effect`. - **Microbenchmark or end to end:** a 3x faster function inside a request that spends 95% of its time on the network changes nothing. Measure end to end before celebrating a micro result. ## 2. Pick the tool Use the ecosystem's own tool if it is installed, because it runs in-process with less overhead:
Details
- Skill name
benchmark-honestly- Repository
- dotslashsimran/simcraft
- Path
skills/coding/benchmark-honestly/SKILL.md- License
- MIT
- Repo updated
- 9 November 2026
FAQ
What does the Benchmark Honestly skill do?
Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance"
How do I install the Benchmark Honestly skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add dotslashsimran/simcraft --skill benchmark-honestly. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Benchmark Honestly popular?
It lives in dotslashsimran/simcraft, which has 517 GitHub stars and 39 forks.
Related skills
API Design Review
517dotslashsimran/simcraft
Review the design of an API surface, meaning an HTTP API (OpenAPI spec or route code), an SDK's public classes and functions, or a CLI. Checks naming and consistency, resource modelling, error design (stable codes, RFC 9457 problem details, request ids), versioning and breaking changes, pagination, filtering, idempotency and retries, rate limits, timeouts, long-running operations, webhooks, auth, partial updates, ID, time and money formats, SDK ergonomics and CLI conventions. Uses scripts to lint OpenAPI files, classify changes between two spec versions as breaking or safe, and dump an SDK's p
Bisect Anything
517dotslashsimran/simcraft
Find the exact commit (or dependency version, config value, or input) that caused a regression by turning it into an automatic pass/fail check and running git bisect run in a separate worktree, then shrink the bad commit to the hunk that matters and explain it. Handles failing tests, changed output, slowdowns, crashes, visual changes, server behaviour, flaky checks, unbuildable commits and merge-heavy history. Use when the user says "what broke this", "find the commit that", "when did this start", "it used to work", "regression", "git bisect", "which version broke it", or "which change made it
Brand Kit
517dotslashsimran/simcraft
Learn a company's brand design from its live website (colours, fonts with the real font files, logo, button and card shapes, texture, image style) and save it as a reusable kit that vox-explainer and image-gen apply automatically. Use FIRST, before any brand or sponsored video, sticker, graphic or asset for a company (e.g. "a video for Devin", "ChatGPT-style stickers", "match their brand"), or when asked to get, learn, extract or match a brand's look.
Code Loop
517dotslashsimran/simcraft
Make a seamless, perfectly looping generative video in code, from a three.js scene or a raw GLSL fragment shader, rendered frame-perfect with jittered anti-aliasing and motion blur, then checked for a visible seam, freezing and blinking. Use when asked for a "seamless loop", a "perfect loop", "generative art video", a "shader loop", "three.js animation to video", to "render a looping animation", a looping GIF from code, or any code-driven animation that has to loop without a visible cut.
Codebase Map
517dotslashsimran/simcraft
Understand an unfamiliar codebase fast, the way senior engineers do. Reads the orientation signals (README, manifests, CI, Dockerfiles, routes, migrations), mines git history for hotspots, hidden coupling and ownership, builds the import graph to find layers, cycles, load-bearing and dead code, then traces one real request or command end to end by actually running it. Writes MAP.md with a one-paragraph purpose, a Mermaid architecture diagram, the traced path as file:line hops, hotspot and coupling tables, a glossary, gotchas and first tasks, plus an optional SVG. Use when the user says "explai
Tree Generator
360TanStack/intent
Use when turning an approved domain_map.yaml and skill_spec.md into a full-library skill tree or explicitly auditing its structure and dependencies. For agreed batches and source-driven guidance updates, use generate-skill instead.
Browse the most popular Claude skills, all software development skills, or write your own with the skills generator.