Psychology › Schools & history › The replication crisis
P-hacking and 'false-positive psychology'
2011· Concept
Well replicatedStrong, repeated evidence.
Trying many analysis choices until a result crosses the p < .05 'significance' line, then reporting only that version.
See it on the Psychology map →What it says
A p-value measures how surprising the data would be if there were no real effect; below .05 traditionally counts as 'significant'. In 2011 Joseph Simmons, Leif Nelson and Uri Simonsohn showed that routine choices, such as stopping data collection when results look good, dropping a measure or adding a control, can push false-positive rates from 5% to over 60%. To prove it, they 'showed' that hearing a Beatles song made people younger. Most p-hacking is not fraud, just unexamined flexibility.
Why it matters
It explains how honest researchers filled journals with false findings. Investors face the same trap: a strategy backtested many ways until it shines usually disappoints with real money.
Think of it like
Like tweaking rent growth, cap rate and exit year until a deal pencils, then showing investors only that pro forma.
The evidence
Status: well replicated. The 2011 simulations are easy to reproduce, surveys show researchers admitted such practices were common, and published p-values cluster suspiciously just below .05.
Open questions
How much of the literature is affected, and whether a stricter threshold (p < .005, proposed in 2018) would help or just move the target.
Connections (3)
▸The replication crisis was driven in part by P-hacking and 'false-positive psychology'Well replicated
The connection
Flexible analysis let researchers find 'significant' results for effects that were not real. Combined with journals publishing mainly positive results and studies too small to detect real effects reliably, the literature filled with exaggerated findings that then failed when others repeated the studies.
Who showed it and when
Named and quantified in 2011–2012 by Joseph Simmons, Leif Nelson and Uri Simonsohn, and in a survey of over 2,000 psychologists by Leslie John and colleagues.
The evidence
In that survey, most respondents admitted at least one questionable practice; replications with locked-in plans show much smaller effects than the originals.
Open questions
How to split the blame among p-hacking, publication bias, small samples and outright fraud such as Diederik Stapel's; their effects are hard to separate.
▸Open science: preregistration and registered reports guards against P-hacking and 'false-positive psychology'Well replicated
The connection
P-hacking needs freedom to try analyses after seeing the data. Preregistration removes that freedom by fixing the sample size, measures and analysis in a public, time-stamped record; later changes must be disclosed, so readers can tell confirmed predictions from after-the-fact exploration.
Who showed it and when
The 2011 'False-Positive Psychology' paper proposed disclosure rules; its authors launched the AsPredicted registry in 2015, alongside the Open Science Framework registry.
The evidence
Comparisons show preregistered studies report fewer positive results and smaller effects, consistent with less p-hacking.
Open questions
Audits find many preregistrations are vague and some deviations go unreported; how much preregistration actually prevents remains under study.
▸Brain imaging: EEG, PET and fMRI is vulnerable to P-hacking and 'false-positive psychology'Well replicated
The connection
An fMRI scan tests tens of thousands of tiny brain units, called voxels, at once, so some will 'light up' by chance unless the statistics correct for it. Flexible analysis choices, small samples and selective reporting add more false positives. Brain imaging became an early, vivid warning case of the p-hacking problem.
Who showed it and when
2009: Craig Bennett's team found 'activity' in a dead salmon shown photos, and Edward Vul flagged 'voodoo correlations'. 2016: Anders Eklund's team in Sweden tested common analysis software.
The evidence
Those papers are published and easy to check; Eklund's team found standard methods could give false-positive rates up to 70% instead of the intended 5%.
Open questions
How many published imaging findings are false; estimates vary, and the field has since moved to larger samples, preregistration and shared data.