Psychology › Schools & history
The replication crisis
2011–present· Concept
Well replicatedStrong, repeated evidence.
The discovery, from 2011 on, that many famous psychology findings shrink or vanish when other scientists repeat the studies.
See it on the Psychology map →What it says
In 2011 a top journal published Daryl Bem's experiments 'showing' that people can sense the future, using standard methods, so the methods themselves looked suspect. That year Diederik Stapel was exposed for faking data, and a paper showed how flexible analysis manufactures false positives. Large repeat studies followed, including the Reproducibility Project and Many Labs. Power posing, ego depletion and social priming largely failed; the marshmallow test and Stanford prison study were weakened or re-examined.
Why it matters
It changed how psychology works and how to read it: treat single striking studies and pop-psychology bestsellers with caution; trust effects confirmed by large, independent repeats.
Think of it like
Like discovering many pro formas used best-case assumptions: when audited actuals came in, returns were often half of projections.
The evidence
Status: well replicated. Systematic projects have repeated hundreds of studies; success rates mostly ran from about one-third to two-thirds, and surviving effects were often about half their original size.
Open questions
Whether reforms have truly raised reliability is still being measured; similar problems appear in medicine, economics and cancer biology.
Inside this (4)
Connections (11)
▸The replication crisis was driven in part by P-hacking and 'false-positive psychology'Well replicated
The connection
Flexible analysis let researchers find 'significant' results for effects that were not real. Combined with journals publishing mainly positive results and studies too small to detect real effects reliably, the literature filled with exaggerated findings that then failed when others repeated the studies.
Who showed it and when
Named and quantified in 2011–2012 by Joseph Simmons, Leif Nelson and Uri Simonsohn, and in a survey of over 2,000 psychologists by Leslie John and colleagues.
The evidence
In that survey, most respondents admitted at least one questionable practice; replications with locked-in plans show much smaller effects than the originals.
Open questions
How to split the blame among p-hacking, publication bias, small samples and outright fraud such as Diederik Stapel's; their effects are hard to separate.
▸Reproducibility Project: under 40% replicated measured the scale of The replication crisisWell replicated
The connection
Before 2015 the crisis was a string of individual failures and arguments. By repeating 100 studies chosen systematically, the project gave it a number: about 36% replicated, with effects about half as large. That figure made headlines worldwide and became the standard shorthand for the problem.
Who showed it and when
Coordinated by Brian Nosek's Center for Open Science in Charlottesville, Virginia; 270 coauthors; published in Science in August 2015.
The evidence
Every protocol, dataset and analysis is public on the Open Science Framework, and original authors reviewed most replication designs in advance.
Open questions
Daniel Gilbert and colleagues argued in 2016 that design differences and chance made the rate look worse than it was; the project's authors disputed this.
▸Open science: preregistration and registered reports is the main response to The replication crisisWell replicated
The connection
Open science targets the crisis's causes. Preregistration ties researchers' hands before data arrive; registered reports remove the bias toward positive results; shared data lets others check the work; and large multi-lab studies supply the statistical power that small studies lacked.
Who showed it and when
Since 2011–2013: the Center for Open Science (Charlottesville, 2013), registered reports at Cortex (2013) and later hundreds of journals, and the Many Labs collaborations.
The evidence
Measurable shifts: registered reports confirm hypotheses far less often than standard papers, and journal badges for open data sharply increased data sharing.
Open questions
Whether reforms raise replication rates. In an irony, a 2023 paper reporting high replicability with these practices was retracted in 2024 over doubts about its own preregistration.
▸WEIRD samples: who psychology studies add a generalizability problem to The replication crisisWell replicated
The connection
The replication crisis asks whether a finding is real; WEIRD samples ask 'real for whom?' An effect repeated many times on American undergraduates may still fail elsewhere. Surprisingly, the Many Labs 2 project, testing 28 effects across 36 countries, found little variation: effects that worked tended to work everywhere, and failures failed everywhere.
Who showed it and when
Joseph Henrich, Steven Heine and Ara Norenzayan, University of British Columbia, 2010; Many Labs 2 published in 2018.
The evidence
Audits of journal samples document the skew; cross-cultural studies show some measures, such as visual illusions and fairness norms, genuinely vary.
Open questions
Which effects differ by culture is unresolved; even Many Labs 2 relied largely on university and online samples rather than truly diverse populations.
▸The replication crisis overturned the hormone claims of Power posingWell replicated
The connection
Power posing was an early, highly visible target. A 2015 study with 200 participants found no hormone or risk-taking effects. In 2016 lead author Dana Carney disowned the findings, citing analysis choices that had inflated them. A 2017 review of 33 power-pose studies found no evidence of a real effect; Cuddy's 2018 rebuttal found support only for feeling more powerful.
Who showed it and when
Eva Ranehill and colleagues in Zurich, 2015; Dana Carney's online statement, September 2016; Joseph Simmons and Uri Simonsohn at the University of Pennsylvania, 2017.
The evidence
Larger replications, some preregistered, found no change in testosterone, cortisol or risk-taking, and an original author withdrew support.
Open questions
How such findings should be communicated: the TED talk reached tens of millions before any large replication, and Cuddy and her critics still disagree about what remains.
▸The replication crisis overturned Ego depletion ('willpower is a muscle')Well replicated
The connection
A 2010 meta-analysis of 198 experiments reported a solid effect, but a 2014 reanalysis correcting for unpublished null results suggested it might be near zero. Direct tests followed: a 2016 registered replication across 23 labs and 2,141 people found almost nothing, and a 2021 effort in 36 labs, involving the theory's proponents, found an effect too small to matter.
Who showed it and when
Evan Carter and Michael McCullough's reanalysis, 2014; Martin Hagger and Nikos Chatzisarantis's multi-lab project, 2016; the 2021 test led by Kathleen Vohs, a longtime Baumeister collaborator.
The evidence
Preregistration fixes methods and analyses in advance, so results cannot be massaged afterward; both large tests were preregistered.
Open questions
Baumeister argued the lab tasks were too mild to drain willpower; critics reply that an effect invisible in 36 labs explains little about daily life.
▸The replication crisis overturned Social priming (the 'slow walking' study)Well replicated
The connection
In 2012 a Belgian team repeated the slow-walking study with infrared timers and found no effect, except when experimenters expected slow walking. Daniel Kahneman then emailed priming researchers warning of a 'train wreck looming' and urging coordinated replications. Large Many Labs projects later failed to reproduce money priming, flag priming and similar effects.
Who showed it and when
Stéphane Doyen and colleagues in Brussels, published in PLOS ONE, 2012; Kahneman's open email, September 2012; Many Labs projects, 2014 and 2018.
The evidence
Failed replications used automated measurement and larger samples. Bargh disputed them vigorously, but no large preregistered study has confirmed the elderly-walking effect.
Open questions
Whether faint cues shift real-world behavior at all, or only by amounts too small for typical studies to detect.
▸The replication crisis re-examined Stanford prison experimentWell replicated
The connection
The crisis taught psychologists to recheck famous studies. The Stanford prison experiment was never a controlled experiment, and in 2018 researchers reading its archives found coaching of guards and staged behavior. Textbooks that had treated it as proof of situational evil began cutting or qualifying it.
Who showed it and when
Stanford University archives; French researcher Thibault Le Texier's 2018 book and journalist Ben Blum's 2018 article; follow-up analyses in American Psychologist in 2019.
The evidence
Original recordings released by Stanford capture the warden urging a reluctant guard to be 'tough.' Zimbardo's 2018 reply disputed the interpretation, not the recordings.
Open questions
Critics raised demand effects as early as 1975, so some ask why textbooks took so long to change, and which other classic demonstrations deserve the same audit.
▸The replication crisis weakened claims of The marshmallow testWell replicated
The connection
In 2018 researchers revisited the test with about 900 children from a US national study, far more diverse than Stanford's preschool. Waiting time still predicted achievement at 15, but half as strongly, and counting family background, early skills and home life cut that by about two-thirds. A 2020 follow-up of Mischel's original children found little link to midlife outcomes.
Who showed it and when
Tyler Watts, Greg Duncan and Haonan Quan, New York University and UC Irvine, 2018; Daniel Benjamin and colleagues' midlife follow-up of the original Stanford participants, 2020.
The evidence
Larger, more representative samples with statistical controls the originals lacked. But the 2018 study was a conceptual replication, with a 7-minute limit rather than Mischel's exact setup.
Open questions
How much the remaining link reflects self-control itself versus the advantages of a stable, well-off home.
▸Dunning–Kruger effect was statistically re-examined during The replication crisisMixed evidence
The connection
During psychology's replication audit, critics re-examined the Dunning–Kruger chart. Statisticians showed that regression to the mean plus the general tendency to rate oneself above average can produce much the same picture, even from random numbers. The effect did not simply fail to replicate; the dispute is over how much of it is real psychology rather than measurement.
Who showed it and when
Critiques began with Joachim Krueger and Ross Mueller in 2002 and revived after 2016, notably in Gilles Gignac and Marcin Zajenkowski's 2020 study of about 930 people.
How solid is it?
Simulations reproduce the classic chart from random data; defenders, such as Joyce Ehrlinger and colleagues (2008), found the gap survived corrections for measurement error.
Open questions
Whether low performers truly lack insight or simply guess near average, and how large the effect is once the statistics are handled properly.
▸The replication crisis shrank the claimed effects of Growth mindset: believing ability can grow (Dweck)Well replicated
The connection
Early studies by Carol Dweck's group suggested that teaching students their ability can grow substantially improved effort and grades. In the replication era, large tests found much smaller effects: a 2018 meta-analysis of 43 interventions found a very small average effect, and a 2019 national US trial found a small grade gain, mostly among lower-achieving students.
Who showed it and when
Meta-analysis by Victoria Sisk, Brooke Macnamara and colleagues (Psychological Science, 2018); trial of over 12,000 ninth-graders by David Yeager, Dweck and colleagues (Nature, 2019).
The evidence
The 2019 trial was preregistered with a nationally representative sample, and the meta-analyses pooled hundreds of studies; both point to small effects.
Open questions
Whether small effects matter at scale in the right schools, as Dweck argues, or are near zero overall, as a 2023 analysis by Macnamara concluded.