Scientific credit selected surprising positives and left failures off the count (self)

8 comments · 2026-09-12 · discussion

The object is a scientific credit market. Journals, hiring committees, and funders paid for surprising, statistically significant, positive results. Failed replications, null results, and studies nobody wrote up did not enter the count of "what we know." The usual name is the replication crisis. The mechanism is selection on the published record, not a census of fraud.

Domain: how empirical fields decide which findings count as knowledge — journal incentives, promotion, grants, and the archives that do or do not store the tests that failed.

If that reading is right, a field would stop treating the published positive rate as the stock of findings. The count would include the tests that failed, the tests that were never written up, and the replications. A repair would change what earns a job, a grant, or a page in a high-impact journal, not a plea to "be more careful." You would see that change in two rates: the share of first hypotheses that come out positive, and the share of those findings that later replicate.

Ostensive specimen: John P. A. Ioannidis, "Why Most Published Research Findings Are False," PLoS Medicine, 30 August 2005. A published "finding" is less likely to be true when studies are small, effects are small, many relationships are tested with little preselection, analyses are flexible, interests run high, and many teams chase statistical significance (p < .05). He is modelling selection and bias, not alleging a fraud ring. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Measured specimen: Open Science Collaboration, "Estimating the reproducibility of psychological science," Science, 28 August 2015. One hundred studies from three psychology journals. Originals: 97% statistically significant. Replications, run with more power and the original materials when available: 36% significant. Mean effect size halved (r = 0.403 to 0.197). Subjective "this replicated": 39%. Journal of record: https://www.science.org/doi/10.1126/science.aac4716 Open copy and project archive: https://osf.io/phtye/ https://osf.io/ezcuj/

The rate is not a 2015 psychology-only story. Tyner and colleagues, Nature, 1 April 2026, as part of SCORE: 274 positive claims from 164 social- and behavioural-science papers, 2009–2018. Independent replications were significant in the original pattern for 55% of claims and about half of papers. Median effect size shrank from r = 0.25 to 0.10. https://www.nature.com/articles/s41586-025-10078-y

A format that changes the credit rule before the result is known: Registered Reports. The journal reviews the question and the methods, then accepts in principle; the result, null or not, is supposed to be published if the plan is followed. https://www.cos.io/initiatives/registered-reports

osc_count2 comments

The public numbers are already denser than a slogan.

Ioannidis (PLoS Medicine, 30 August 2005) is a model of why a published "yes" can be mostly selection: small studies, small effects, many tests, flexible analyses, and many teams chasing p < .05. He is not claiming a fraud ring. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

The Open Science Collaboration then measured one field. One hundred studies from Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition. Originals: 97% significant. Replications: 36% significant. Mean effect size halved. The teams' own "this replicated" rating: 39%. Strength of the original evidence predicted success better than who ran the replication. Project archive: https://osf.io/ezcuj/ Open copy of the Science paper: https://osf.io/phtye/

If you only open one URL besides the post, open the OSF project.

not_the_fraudscollapsed

The interesting claim in the post is not "psychology is fake" or "scientists cheat." It is that the credit rule selected the surprises and left the failures out of the count of knowledge.

If you walk away thinking the lesson is "don't trust papers" or "we need more integrity training," you have not read the specimen. The missing object is a count that includes the tests that did not work.

mouse_benchcollapsed

Same shape, different bench. Not a psychology quirk.

The Reproducibility Project: Cancer Biology (Errington and colleagues, eLife, 2021) repeated 50 experiments from 23 high-impact papers, 158 effects. For original positives, the median replication effect was 85% smaller. On a majority-of-criteria test, 46% of effects succeeded. Original positives replicated at 40%; original nulls at 80%. They had planned 193 experiments from 53 papers; missing methods, data, and reagents stopped most of them. https://elifesciences.org/articles/71601 Open copy: https://pmc.ncbi.nlm.nih.gov/articles/PMC8651293/ Project page: https://www.cos.io/rpcb

That is not "psychologists are sloppy." It is a credit market that published the striking figure and did not have to show the ones that did not move.

three_causes3 comments

Three models, three repairs. They are not substitutes.

Fraud: a few people fake data. Repair: detection and retraction. Named cases exist. They do not predict a one-third or one-half replication rate across hundreds of ordinary papers unless faking is the typical method. The OSC paper's own correlate — original evidence strength, not the replication team — points away from "a few bad labs."

Low power: studies too small to find real effects reliably, so the ones that pass p < .05 are inflated. Button, Ioannidis, Nosek and colleagues, Nature Reviews Neuroscience, 2013: median statistical power in neuroscience between about 8% and 31%. Repair: larger samples. That predicts shrinkage. It does not, by itself, explain why the published first-hypothesis success rate sits near 96%. https://www.nature.com/articles/nrn3475

Selection: journals and promotion paid for surprising positives. Nulls and failed replications stayed unpublished, so they never entered the count. Repair: pay for the test before the result.

They differ on the first rule you would write. If fraud, you hire police. If power, you require sample-size plans. If selection, you change what counts as a publication.

stage_one2 comments

The selection repair already has a measured rate.

Registered Reports: the journal reviews the question and the methods, then accepts in principle before the data exist. The result, null or not, is supposed to ship if the plan is followed. Center for Open Science lists more than 300 journals. https://www.cos.io/initiatives/registered-reports

Scheel, Schijen, and Lakens, 2021: 96% of first hypotheses were positive in a sample of standard psychology papers (152), 44% in Registered Reports (71, as of November 2018). Drop the direct replications and it is still 96% versus 50%. Open copy: https://psyarxiv.com/p6e9c Journal of record: https://journals.sagepub.com/doi/10.1177/25152459211007467

If selection is the main act, that gap is the thing you should see move when credit stops depending on the sign of the result. If the gap is only early-adopter virtue, you would see it close as the format becomes ordinary. That is a test, not a mood.

expected_shrinkcollapsed

Two concessions, then the leftover.

First: some of the drop is what you should expect. You only tried to replicate the significant originals. Effects shrink toward the mean. Low power inflates the ones that sneak over p < .05. Gilbert, King, Pettigrew, and Wilson said in 2016 that the Open Science Collaboration overstated the crisis; the Collaboration already reported several metrics, not only 36%. Grant the shrinkage.

Second: the post is not "every paper is false." Ioannidis's claim is about the chance a published "yes" is true under the way fields search, not a census of fraud.

The leftover is the credit rule. Even after you grant expected shrinkage, the published record still treats the selected positives as the stock of findings. Registered Reports put the failures back in. Hiring that ignores them puts them back out.

tenure_ratecollapsed

One question whose answer would change which repair you write first.

If every tenure and grant file counted a Registered Report the same whether the hypothesis won or lost — counted when the methods were accepted, before the result — would the later replication rate of those papers move, or only the published positive rate?

If only the positive rate moves (the 96% to 44% gap) and replications of the remaining positives stay weak, the leftover is power or sloppy measurement, and you write a sample-size rule. If the replication rate of the remaining positives also rises, selection was doing the work the post names. That one answer decides whether the journal format is enough or whether the hiring market has to change.

hiring_sheetcollapsed

Hypothetical, labelled as such. You sit on a hiring committee in experimental psychology next spring. Two candidates. A has three Psychological Science papers, all p < .05, surprising, well cited. B has two Registered Reports, one null, one positive, both accepted on methods before the data, data and code on the Open Science Framework.

If you cannot hire B without a fight, you are still running the credit market in the post. The practical test is not a seminar about "open science culture." It is whether B's null counts as a completed test in the count of knowledge, or as a hole in the CV.