Improving what is published: Toward an evidence-based framework for manuscript review.

Abstract

Recent findings indicate that many adults in the United States are skeptical regarding scientific evidence. Some of this skepticism is rooted in political partisanship, but distrust of research findings exists on both ends of the political spectrum. One way to begin to restore faith in the validity and utility of scientific research is to take a closer look at the effectiveness of the manuscript-evaluation process. To that end, this article examines current findings regarding the strengths and limitations of manuscript review and identifies ways that the process can be improved. To move toward a more evidence-based framework, we must take into account two psychological dynamics that shape the review process. First, manuscript review is what is known in psychology as a "signal-detection task": Reviewers and editors attempt to distinguish signals (publishable manuscripts) from noise (manuscripts that do not warrant publication). Second, manuscript review involves a complex social interaction among authors, reviewers, and editors, each of whom brings particular goals to the endeavor and each of whom must communicate with multiple audiences simultaneously to achieve those goals. The practical implications of the findings in this area are discussed, and preliminary steps for moving toward a more evidence-based approach to manuscript review are outlined.

Promoting equity in the peer review process of journal publication

“…We, thus, encourage the science education journal community to (i) systematically probe their peer-review process for diversity, equity, and bias, (ii) expand the diversity of journal editors and editorial board members as guided by their systematic probe, (iii) explicitly communicate being welcoming of equity focused research, and (iv) put in place transparent systems to address unprofessional reviewer comments. We also encourage members of the community who teach doctoral courses to explicitly train students to use the debiasing strategies proposed by Bornstein (2021) to provide fair, polite, and constructive feedback to all authors. Collectively, these actions can initiate the normalization of a journal peer review process focusing on the merits of diverse scholarships and de-emphasizes the interactions among the identities of reviewers', editors', and authors' known to lead to social bias in the peer review process.…”

“…Training has some small impact on improving quality of review, but its effect wanes over time (Schroter et al, 2004). Bornstein (2021) recommends both editors and reviewers use “debiasing strategies” for both manuscript reviewers and editors including (1) ongoing training in research and theories on reasoning and distortion in human decision making as well as on how specific forms of bias influence manuscript evaluation, which is a deeper and more long term approach than any current training available from any of the journal's reviewed, (2) self‐monitor for fatigue, distraction, cognitive overload when evaluating manuscripts and composing feedback, and (3) mindfulness regarding self‐presentation goals (i.e., ensure they are giving professional, politely toned, constructive feedback). Bornstein (2021) also recommends formalizing this training in doctoral programs so junior scholars enter the profession aware of how to give fair and useful reviews.…”

EDI and open access: How JACMP is the future of ethical publishing—A tale in two parts

“…The confirmation bias can influence the assessment of research quality if the reviewer thinks the data supports their own beliefs. There are many studies evaluating the impact of bias and the effectiveness of peer review 15–17 …”

“…There are many studies evaluating the impact of bias and the effectiveness of peer review. 15 , 16 , 17 …”

“…evaluating the impact of bias and the effectiveness of peer review. [15][16][17] Finally, it is important to think about the readers of the journals, evaluating the reach and impact of the journal. There are barriers that could prevent equal access to these scientific publications, such as the cost of a subscription.…”