Novelty versus Replicability: Virtues and Vices in the Reward System of Science

Romero, Felipe

doi:10.1086/694005

Cited by 41 publications

(29 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…15 The culture of 'publish or perish' is widely blamed for poor standards and practices (John et al 2012;Simmons et al 2011)-'questionable research practices' as they have become known-along with journal editors' biases towards positive over negative results (see above) and in favour of papers meeting inappropriate aesthetic standards (Giner-Sorolla 2012). There are powerful incen-tives for researchers to produce interesting positive results and so it would be no surprise that some researchers cut corners, leading to outcomes that cannot later be replicated (Romero 2017) as well as disincentivizing replication itself.…”

Section: Bias Questionable Research Practices and Fraudmentioning

confidence: 99%

Understanding the Replication Crisis as a Base Rate Fallacy

Bird

2021

The British Journal for the Philosophy of Science

View full text Add to dashboard Cite

The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the field of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science. 1.1The replication crisis1.2The base rate fallacy2From the Base Rate Fallacy to the Replication Crisis3Explaining the `Replication Crisis: Low π and Non-negligible α3.1Science with low π3.2The value of α: Type-I errors4Other Explanations of the Crisis4.1Low statistical power4.2Publication bias4.3Bias, questionable research practices, and fraud5What Is to Be Done?5.1Quietism: This is the nature of science5.2Higher quality hypotheses5.3Decrease α6Conclusion

show abstract

Section: Bias Questionable Research Practices and Fraudmentioning

confidence: 99%

Understanding the Replication Crisis as a Base Rate Fallacy

Bird

2021

The British Journal for the Philosophy of Science

View full text Add to dashboard Cite

show abstract

“…Although the finding persuaded very few scientists, the controversy engendered mistrust in the ways psychologists conduct their experiments because Bem used procedures and statistical tools that many social psychologists use. (See Romero, , for a discussion. ) Amgen and Bayer Healthcare reports.…”

Section: What Is the Replicability Crisis? History And Evidencementioning

confidence: 99%

Philosophy of science and the replicability crisis

Romero

2019

Philosophy Compass

Self Cite

View full text Add to dashboard Cite

Replicability is widely taken to ground the epistemic authority of science. However, in recent years, important published findings in the social, behavioral, and biomedical sciences have failed to replicate, suggesting that these fields are facing a “replicability crisis.” For philosophers, the crisis should not be taken as bad news but as an opportunity to do work on several fronts, including conceptual analysis, history and philosophy of science, research ethics, and social epistemology. This article introduces philosophers to these discussions. First, I discuss precedents and evidence for the crisis. Second, I discuss methodological, statistical, and social‐structural factors that have contributed to the crisis. Third, I focus on the philosophical issues raised by the crisis. Finally, I discuss several proposals for solutions and highlight the gaps that philosophers could focus on.

show abstract

“…We identify three types of reforms that can be regarded as complementary rather than mutually exclusive. First, social reforms, which are inspired by the prevalence of questionable research practices ("QRPs": Simmons et al 2011) and more generally, the adverse effects of social and structural factors in science (Bakker et al 2012;Nuijten et al 2016;Romero 2017). Social reforms include educating researchers about statistical cognition and methodology (Schmidt 1996;Lakens 2019), but also creating greater incentives for replication work-for example by publishing and co-citing replications alongside original studies (Koole and Lakens 2012) or establishing a separate reward system for confirmatory research (Romero 2018).…”

Section: Introductionmentioning

confidence: 99%

Scientific self-correction: the Bayesian way

Romero

Sprenger

2020

Synthese

Self Cite

View full text Add to dashboard Cite

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure-replacing null hypothesis significance testing (NHST) with Bayesian inference-in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study.

show abstract

Novelty versus Replicability: Virtues and Vices in the Reward System of Science

Cited by 41 publications

References 37 publications

Understanding the Replication Crisis as a Base Rate Fallacy

Understanding the Replication Crisis as a Base Rate Fallacy

Philosophy of science and the replicability crisis

Scientific self-correction: the Bayesian way

Contact Info

Product

Resources

About