Both Questionable and Open Research Practices Are Prevalent in Education Research

Makel, Matthew C.; Hodges, Jaret; Cook, Bryan G.; Plucker, Jonathan A.

doi:10.3102/0013189x211001356

Cited by 61 publications

(83 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The lower rates reported in Krishna et al (2018) could also be due to the focus on the undergraduate thesis specifically: in our study, students were reporting on broader research involvement, where there could be more opportunity for QRPs. Further, despite some methodological differences, the rates of self-reported QRP use from this study are broadly comparable to previous surveys of non-student researchers in psychology and related disciplines (Agnoli et al, 2017;Héroux et al, 2017;Janke et al, 2019;John et al, 2012), although some studies report slightly higher (Artino et al, 2019;Chin, 2021;Makel et al, 2021) and lower (Fiedler & Schwarz, 2016;Fox et al, 2018;Rabelo et al, 2020;Wolff et al, 2018) estimates (Table 1). It appears that rates reported by students in the current study are more similar to previous research with non-student researcher samples than to student samples.…”

Section: Discussionsupporting

confidence: 82%

“I know it's bad but I have been pressured into it”: Questionable research practices among psychology students in Canada

Moran¹,

Richard²,

Wilson³

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Questionable research practices (QRPs) have been identified as a driving force of the replication crisis in the field of psychological science. The aim of this study was to assess the frequency of QRP use among psychology students in Canadian universities, and to better understand reasons and motivations for QRP use.Method: Participants were psychology students attending Canadian universities and were recruited via online advertising and university email invitations to complete a bilingual survey. Respondents were asked how often they and others engaged in seven QRPs. They were also asked to estimate the proportion of psychology research impacted by each QRP and how acceptable they found each QRP. Data were collected through Likert-scale survey items and open-ended text responses between May 2020 and January 2021, and was analyzed using descriptive statistics and thematic analysis. Results: 425 psychology students completed the survey. The sample consisted of 40% undergraduate students, 59% graduate students and 1% post-doctoral fellows. Overall, 64% of participants reported using at least one QRP, while 79% reported having observed others engaging in at least one QRP. The most frequently reported QRPs were p-hacking (46%), not submitting null results for publication (31%), excluding outcome measures (30%), and hypothesizing after results are known (27%). These QRPs were also the most frequently observed in others, estimated to be the most prevalent in the field, and rated as the most acceptable. Qualitative findings show that students reported that pressures to publish motivated their QRP use, with some reporting that certain QRPs are justifiable in some cases (e.g., in the case of exploratory research). Students also reported that QRPs contribute to the replication crisis and to publication bias and offered several alternatives and solutions to engaging in QRPs, such as gaining familiarity with open science practices. Conclusions: Most Canadian psychology students in this sample report using QRPs, which is unsurprising since they observe such practices in their research environment and estimate that they are prevalent. In contrast, most students believe that QRPs are not acceptable. The results of this study highlight the need to examine the pedagogical standards and cultural norms in academia that may promote or normalize QRPs in psychological science, to improve the quality and replicability of research in this field.

show abstract

Section: Discussionsupporting

confidence: 82%

“I know it's bad but I have been pressured into it”: Questionable research practices among psychology students in Canada

Moran¹,

Richard²,

Wilson³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These include presenting unexpected results as having been predicted a priori (hypothesizing after results are known [HARKing]; Kerr, 1998) and exploiting flexibility in data analysis to obtain statistically significant results (p-hacking; Simmons et al, 2011). Evidence for both categories of bias exists: Publication bias has been observed in peer review (Atkinson et al, 1982;Mahoney, 1977) and in longitudinal data from a National Science Foundation grant program that found a filedrawering effect for studies with negative results (Franco et al, 2014(Franco et al, , 2016, and QRPs have been admitted by scientists in several survey studies (Agnoli et al, 2017;Fiedler & Schwarz, 2016;Fraser et al, 2018;John et al, 2012;Makel et al, 2021). Some authors have argued that negative results are often uninformative or the result of low-quality research and should not be published at the same rate as positive results to avoid cluttering the literature (e.g., Baumeister, 2016;Cleophas & Cleophas, 1999;Mitchell, 2014).…”

Section: A Biased Literaturementioning

confidence: 99%

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Scheel

Schijen

Lakens

2021

Advances in Methods and Practices in Psychological Science

208

143

View full text Add to dashboard Cite

Selectively publishing results that support the tested hypotheses (“positive” results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports (RRs), peer review and the decision to publish take place before results are known. We compared the results in published RRs ( N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature ( N = 152) in psychology. Analyzing the first hypothesis of each article, we found 96% positive results in standard reports but only 44% positive results in RRs. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type I error inflation in the RR literature.

show abstract

“…Questionable research practices have been defined 5 as including: failing to report all of a study's dependent measures, failing to report all of a study's conditions, collecting more or less data during analysis to support a hypothesis, rounding p values, claiming to have expected unusual findings (HARKing), claiming that demographics have no impact on data, and outright falsification of data. These practices have recently been found to be prevalent among education researchers 23 . It is important here to ensure that the push to open research practices does not turn into a witch hunt, while there is little peer-reviewed literature on this, there are some concerning reports that the open research movement may negatively impact researchers careers 24 .…”

Section: Does Veted Need Open Science?mentioning

confidence: 99%

Open Science for Veterinary Education

MacKay¹

2021

Preprint

View full text Add to dashboard Cite

“Open Science” encourages researchers to improve the reliability, repeatability and reproducibility of results. While stemming from the reproducibility crisis in psychology, open science is relevant for all fields. In education research these practices include pre-registering experimental designs, sharing analyses where appropriate, and pre-printing research report. How can veterinary education implement these lessons? In this short report accompanying the ‘Open Science for VetEd’ workshop, hosted at the VetEd 2021 Annual Conference, I discuss what opportunities open science presents for veterinary education researchers.

show abstract

Both Questionable and Open Research Practices Are Prevalent in Education Research

Cited by 61 publications

References 45 publications

“I know it's bad but I have been pressured into it”: Questionable research practices among psychology students in Canada

“I know it's bad but I have been pressured into it”: Questionable research practices among psychology students in Canada

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Open Science for Veterinary Education

Contact Info

Product

Resources

About