The 0.05 Fault Line: Inside American Science's Longest Running Methodological Feud

In 1925, a British statistician named Ronald Fisher published a table in a book about crop experiments. From that table grew a convention so entrenched that most American doctoral students today cannot imagine conducting research without it: the p-value, and the magic threshold of 0.05. Nearly a century later, the convention is under siege — and the siege has exposed a fault line running through every department of psychology, medicine, and economics in the country.

A Threshold That Was Never a Law

Fisher himself never intended 0.05 as a commandment. It was, by most historical accounts, a convenient rounding of the evidence standard for agricultural field trials — a rough gauge of when a result was interesting enough to repeat. Yet by mid-century, the threshold had hardened into bureaucratic doctrine. Grant applications at the National Institutes of Health, clinical trial protocols reviewed by the FDA, and dissertation defenses from Ann Arbor to Austin all came to treat p < 0.05 as the border between science and noise.

The consequences of that hardening are now familiar. Researchers learn, often implicitly, that a p-value of 0.049 is publishable while 0.051 is a career inconvenience — a distinction with no statistical meaning whatsoever. The practice of "p-hacking," in which analysts flexibly choose measures, subgroups, or stopping rules until the number dips below the line, became a rational response to an irrational incentive structure.

The Reformers and the Reluctant

In 2016, the American Statistical Association took the unusual step of issuing a formal statement on p-values — a document prompted, in part, by the reproducibility crisis that had swept through social psychology and biomedicine. Two years later, a group of prominent statisticians called for abandoning the term "statistically significant" altogether, publishing their argument in a special issue of The American Statistician.

Not everyone is persuaded. Clinicians who run large randomized trials argue that thresholds, whatever their flaws, provide a shared decision standard when lives and regulatory approvals hang in the balance. "You cannot run a drug approval process on the phrase 'the weight of evidence suggests,'" one biostatistician told this publication. "At some point, someone has to say yes or no."

Epidemiology has moved furthest. Leading journals in that field now discourage or ban the use of significance asterisks, asking authors to report effect sizes and confidence intervals instead. Psychology has embraced pre-registration, a reform that constrains analytic flexibility but does not, by itself, eliminate the cutoff. Meanwhile, machine learning research — a field with no Fisherian tradition to shed — has largely skipped the debate entirely, relying on held-out test sets rather than hypothesis tests to certify performance.

What Comes After the Cutoff

The most concrete proposals share a common structure: replace a binary verdict with a graded estimate. Confidence intervals, which communicate both the size and the uncertainty of an effect, are the most popular alternative. Bayes factors, which quantify how strongly data shift prior beliefs, have gained traction in cognitive science. Some methodologists advocate for "estimation thinking" — an approach in which researchers report what the effect probably is, rather than whether it cleared an arbitrary bar.

Critics of reform raise a practical worry: thresholds are legible. A tenure committee in Ohio or a hospital pharmacy board in Minnesota can process "p < 0.05" in seconds. Whether they can process a posterior distribution is an open question, and one that reformers have been slow to answer.

There is also a deeper cultural obstacle. The 0.05 threshold did not survive because it was statistically sound; it survived because it simplified judgment. Any replacement will have to be simpler than the problem it solves.

The Stakes of Inertia

The cost of the status quo is not abstract. Meta-analyses of the published literature suggest that results hovering just below significance are dramatically overrepresented in journals — a pattern consistent with widespread threshold gaming. Findings that fail the test are filed away, and the scientific record becomes a curated highlight reel rather than an honest ledger.

For a journal devoted to how science knows what it knows, the p-value controversy is less a technical quarrel than a test case: can a profession built on evidence summon the evidence to reform its own habits? The answer, so far, is that American science is changing — field by field, journal by journal, at the deliberate pace of institutions that have learned to distrust revolutions, including their own.