Alterlab Test Selection Guard

Enforces statistical-test selection as a discipline, holding the Iron Law NO TEST CHOSEN AFTER SEEING THE P-VALUE: routes the choice through a fixed decision tree (outcome type -> groups -> paired? -> normality) terminating in named tests (t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis, Wilcoxon, Friedman, chi-square, Pearson/Spearman, regression), gates interpretation behind a mandatory Shapiro-Wilk/Levene/linearity assumption check, blocks test-shopping with an Excuse-vs-Reality table and Red-Flags-STOP list, and applies a 3+-tests escalation gate forcing Bonferroni/FDR correction or an explo

87 stars 18 forksTesting & QA Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add AlterLab-IEU/AlterLab-Academic-Skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add AlterLab-IEU/AlterLab-Academic-Skills --skill alterlab-test-selection-guard
Manual install
git clone --depth 1 https://github.com/AlterLab-IEU/AlterLab-Academic-Skills.git /tmp/AlterLab-Academic-Skills
mkdir -p ~/.claude/skills
cp -r /tmp/AlterLab-Academic-Skills/skills/methodology/alterlab-test-selection-guard ~/.claude/skills/alterlab-test-selection-guard

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Test-Selection Guard — Iron Law: No Test Chosen After Seeing the P-Value

**Skill type: DISCIPLINE-ENFORCING.** This is a thin discipline layer, not a stats
engine. It does not run tests, fit models, or compute power — it forces the *order* of
operations so the test is chosen by the data's shape, never by the result it produces.
For the actual computation, hand off to the skills below.

> **REQUIRED BACKGROUND** — this skill orchestrates, it does not reimplement:
> - **`alterlab-statistical-analysis`** — runs the chosen test, the assumption checks
>   (its `assumption_checks.py`: Shapiro-Wilk, Levene, Q-Q), power, and APA reporting.
> - **`alterlab-statsmodels`** — fits the model (OLS/GLM/mixed/ARIMA) once the test is fixed.
> - **`alterlab-scientific-thinking`** — judges design validity, biases, confounders upstream.
> - **`alterlab-preregistration-discipline`** — the frozen analysis plan this gate presumes.
>
> When the user needs *execution*, route there. This skill's whole job is what happens
> **before** the first `scipy.stats` call.

---

## The Iron Law

```
NO TEST CHOSEN AFTER SEEING THE P-VALUE.
```

The test is a function of the **research question and the data's structure** — outcome
type, number of groups, pairing, and assumption checks — fixed *before* any p-value is
visible. Choosing or switching a test in response to its significance is p-hacking. It
inflates the Type I error rate by an unknown amount and makes the reported p-value a lie.
Read the full skill on GitHub

Details

Skill name
alterlab-test-selection-guard
Path
skills/methodology/alterlab-test-selection-guard/SKILL.md
License
MIT
Repo updated
11 November 2026

FAQ

What does the Alterlab Test Selection Guard skill do?

Enforces statistical-test selection as a discipline, holding the Iron Law NO TEST CHOSEN AFTER SEEING THE P-VALUE: routes the choice through a fixed decision tree (outcome type -> groups -> paired? -> normality) terminating in named tests (t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis, Wilcoxon, Friedman, chi-square, Pearson/Spearman, regression), gates interpretation behind a mandatory Shapiro-Wilk/Levene/linearity assumption check, blocks test-shopping with an Excuse-vs-Reality table and Red-Flags-STOP list, and applies a 3+-tests escalation gate forcing Bonferroni/FDR correction or an explo

How do I install the Alterlab Test Selection Guard skill?

Claude Code plugin: /plugin marketplace add AlterLab-IEU/AlterLab-Academic-Skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Alterlab Test Selection Guard popular?

It lives in AlterLab-IEU/AlterLab-Academic-Skills, which has 87 GitHub stars and 18 forks, and is included in 1 curated skill list.

Related skills

Alterlab Abm Mesa

87

AlterLab-IEU/AlterLab-Academic-Skills

Builds agent-based models of social systems with Mesa 3 — the current AgentSet API (model.agents.shuffle_do('step'), auto-assigned unique_id, mandatory super().__init__(rng=...)), cell spaces (mesa.discrete_space OrthogonalMooreGrid / classic mesa.space grids), the DataCollector, batch_run parameter sweeps, and SolaraViz — for emergence, segregation, diffusion, opinion dynamics, and cooperation models. It uses the Mesa 3.x API (the old mesa.time schedulers like RandomActivation are removed) and treats the model as a generative theory to be validated, not just run. Use when the request mentions

Research & ScienceIn 1 list18

Alterlab Academic Career

87

AlterLab-IEU/AlterLab-Academic-Skills

Prepares academic career documents and professional-development materials for faculty and researchers. Use when drafting academic CVs, research statements, teaching philosophies, diversity statements, faculty-position cover letters, tenure dossiers, promotion narratives, academic portfolios, or mentorship statements, and when planning conference-networking strategy, building academic web presence, setting up ORCID profiles, or understanding impact metrics. Part of the AlterLab Academic Skills suite.

Research & ScienceIn 1 list18

Alterlab Accreditation Aol

87

AlterLab-IEU/AlterLab-Academic-Skills

Scaffolds program-level Assurance-of-Learning (AoL) documentation for AACSB (Standard 5 in the 2020 standards and the 2026 Global Standards) and ABET (Criterion 3 Student Outcomes, Criterion 4 Continuous Improvement) accreditation — program learning outcomes / competency goals, curriculum-to-outcome mapping matrices, direct- and indirect-assessment plans, rubric design, and closing-the-loop continuous-improvement narratives — and validates the structure of an outcome-mapping matrix with scripts/aol_matrix.py. Use when the user needs AACSB or ABET assurance-of-learning material, a program-learn

Learning & EducationIn 1 list18

Test Crewai Project

49

crewAIInc/skills

Deterministic, offline testing of CrewAI crews and flows with pytest: a stub BaseLLM (shipped in references/stub_llm.py) that drives text, structured output and tool calls with no API key, asserting on the prompts crewai built, testing guardrails and flow routing, what `crewai test` really does, event listeners for debugging, and the exact tracing/telemetry env var values. Use when writing tests for a crewai project, mocking or stubbing the LLM, writing a custom BaseLLM, seeing `TypeError: ... call() got an unexpected keyword argument 'from_task'`, `ValueError: OPENAI_API_KEY is required` in C

Testing & QA24

Alterlab Preregistration Discipline

87

AlterLab-IEU/AlterLab-Academic-Skills

Enforces pre-registration discipline with the Iron Law NO DATA ANALYSIS WITHOUT A PRE-REGISTERED ANALYSIS PLAN FIRST, a spirit-vs-letter line, an Excuse-vs-Reality rationalization table, and a Red-Flags-STOP list (HARKing, optional stopping, post-hoc covariates, outlier-dropping, test-shopping). Runs a PLAN/COLLECT/CONFIRM/EXPLORE workflow that freezes hypotheses, tests, exclusions, and stopping rules before data, then forces unplanned findings to be labeled exploratory (their p-values lose confirmatory status, per COS confirmatory/exploratory model). Orchestrates, not replaces, alterlab-open-

Testing & QAIn 1 list18

A B Testing

79

LeadMagic/gtm-skills

Design statistically sound GTM experiments for copy, channels, landing pages, sequencing, pricing, and funnel conversion. Produces hypothesis, sample-size logic, success metrics, test plan, analysis method, and scale/stop/kill recommendation. Use when planning A/B tests, split tests, or experiment roadmaps.

Testing & QAIn 1 list18

Browse the most popular Claude skills, all testing & qa skills, or write your own with the skills generator.