How do academics assess the results of multiple experiments?

Akker, Olmo R. van den; Ld, Alvarez; Bakker, M.P. de; Wicherts, Jelte M.; Assen, Marcel A. L. M. van

doi:10.31234/osf.io/xyks4

Cited by 9 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proportion of research questions with more true results. Scientists sometimes assess evidence for research questions using heuristic tallies of positive and negative results 38 . As such, the proportion of questions with more true than false results is a useful metric for evaluating the proportion of questions for which scientists will acquire accurate beliefs.…”

Section: Resultsmentioning

confidence: 99%

Competition for priority harms the reliability of science, but reforms can help

2021

View full text Add to dashboard Cite

cademic science is a culturally evolved social institution with formal rules, norms and conventions. However, in recent years, scientists have begun to examine the utility of even longstanding characteristics of this institution 1-3 . For example, it is now widely recognized that preferentially valuing positive over negative results can generate publication bias, which distorts the published literature 4,5 ; evaluating scientists based on their number of publications can cause a myopic focus on productivity at the expense of rigour 6 ; and rewarding scientists based on the prestige of the journal in which they publish may incentivize scientists to present their work in an overly positive light, submit low-quality papers to high-impact journals and engage in other questionable research practices [7][8][9][10][11][12] .The priority rule is a particularly longstanding scientific norm, in which individuals who are first to make discoveries receive disproportionate credit relative to all other individuals who provide solutions to the same problem 13,14 . Famously, Charles Darwin was motivated to publish his writings on evolution by means of natural selection in part because of a concern that he would lose priority to Alfred Russel Wallace, who had developed a similar idea. In his famous letter to Charles Lyell, Darwin proclaimed "I rather hate the idea of writing for priority, yet I certainly should be vexed if any one were to publish my doctrines before me" 15 . Rewards for priority take on various forms, including eponymy (that is, naming a scientific discovery after the scientist who discovered it), financial prizes (for example, the Nobel prize), an increased probability of publishing in high-impact journals, and better professional positions and speaking engagements 3,13,16,17 . Little research explicitly documents the career repercussions of losing a priority race (that is, getting scooped). However, one survey of physical and biological scientists found that over 60% of scientists reported being scooped at some point in their careers 18 , and a recent study among structural biologists found that scooped papers received 28% fewer citations and were 18% less likely to appear in a top-ten journal 19 . This suggests that scientists have significant incentives to compete over priority of discovery.Given its role as a major incentive, how does rewarding priority of discovery affect scientific inquiry? Rewards for priority can

show abstract

Section: Resultsmentioning

confidence: 99%

Competition for priority harms the reliability of science, but reforms can help

2021

View full text Add to dashboard Cite

show abstract

“…This, in turn, reduces the chances of obtaining false-positive results ( Wicherts et al, 2016 ), thus increasing replicability rates ( Munafò, 2016 ; Munafò et al, 2017 ). Given that replications significantly contribute to trust in psychological theories ( van den Akker et al, 2018 ) and that low replicability has been shown to impair public trust in science ( Anvari and Lakens, 2018 ; Chopik et al, 2018 ; Hendriks et al, 2020 ; Wingen et al, 2020 ), increased trust is thus a likely result of OSPs. It should, however, be noted that some of these assumptions cannot be tested empirically, and that at least one study found that low replicability does not have much of a detrimental effect on public perceptions of science ( Mede et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

Open science and public trust in science: Results from two studies

Rosman¹,

Bošnjak

Silber

et al. 2022

Public Underst Sci

View full text Add to dashboard Cite

In two studies, we examined whether open science practices, such as making materials, data, and code of a study openly accessible, positively affect public trust in science. Furthermore, we investigated whether the potential trust-damaging effects of research being funded privately (e.g. by a commercial enterprise) may be buffered by such practices. After preregistering six hypotheses, we conducted a survey study (Study 1; N = 504) and an experimental study (Study 2; N = 588) in two German general population samples. In both studies, we found evidence for the positive effects of open science practices on trust, though it should be noted that in Study 2, results were more inconsistent. We did not however find evidence for the aforementioned buffering effect. We conclude that while open science practices may contribute to increasing trust in science, the importance of making use of open science practices visible should not be underestimated.

show abstract

“…Response rates and invitation procedure. Response rates were rather low in many studies that relied on researchers as sample (e.g., [9,26,38,39,53,54,60]). Based on these studies and based on the insights from our pilot study (see section pilot study), we anticipated a response rate of around 10%.…”

Section: Plos Onementioning

confidence: 99%

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Spitzer

Mueller²

2023

PLoS ONE

View full text Add to dashboard Cite

Background Preregistration, the open science practice of specifying and registering details of a planned study prior to knowing the data, increases the transparency and reproducibility of research. Large-scale replication attempts for psychological results yielded shockingly low success rates and contributed to an increasing demand for open science practices among psychologists. However, preregistering one’s studies is still not the norm in the field. Here, we conducted a study to explore possible reasons for this discrepancy. Methods In a mixed-methods approach, we conducted an online survey assessing attitudes, motivations, and perceived obstacles with respect to preregistration. Respondents (N = 289) were psychological researchers that were recruited through their publications on Web of Science, PubMed, PSYNDEX, and PsycInfo, and preregistrations on OSF Registries. Based on the theory of planned behavior, we predicted that positive attitudes (moderated by the perceived importance of preregistration) as well as a favorable subjective norm and higher perceived behavioral control positively influence researchers’ intention to preregister (directional hypothesis 1). Furthermore, we expected an influence of research experience on attitudes and perceived motivations and obstacles regarding preregistration (non-directional hypothesis 2). We analyzed these hypotheses with multiple regression models and included preregistration experience as a control variable. Results Researchers’ attitudes, subjective norms, perceived behavioral control, and the perceived importance of preregistration significantly predicted researchers’ intention to use preregistration in the future (see hypothesis 1). Research experience influenced both researchers’ attitudes and their perception of motivations to preregister, but not the perception of obstacles (see hypothesis 2). Descriptive reports on researchers’ attitudes, motivations and obstacles regarding preregistration are provided. Discussion Many researchers had already preregistered and had a rather positive attitude toward preregistration. Nevertheless, several obstacles were identified that may be addressed to improve and foster preregistration.

show abstract

How do academics assess the results of multiple experiments?

Cited by 9 publications

References 0 publications

Competition for priority harms the reliability of science, but reforms can help

Competition for priority harms the reliability of science, but reforms can help

Open science and public trust in science: Results from two studies

Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Contact Info

Product

Resources

About