Quantifying the Volunteer Effort of Scientific Peer Reviewing

Abstract

A survey of reviewers at Monthly Weather Review shows how they perform their reviews and allows a calculation of how much volunteer effort goes into producing a scientific journal. W e are always amazed by the huge effort and the degree of seriousness that peer reviewers bring to the task of reading submitted manuscripts and writing reviews-a voluntary task that takes time away from their own research, other work responsibilities, and personal time. By providing written comments on manuscripts that have been submitted to journals for publication, reviewers improve the quality of manuscripts that eventually are published or they recommend rejection. In turn, reviewers benefit from having the chance to read the latest research before it is formally published, influence research that may ultimately be published, demonstrate the ability to perform high-quality reviews and become an associate editor or editor eventually, develop contacts and collaborations, and improve their own writing and science. The workload on reviewers can be tremendous. Apart from anecdotal evidence, however, quantitative measurement of the time spent on the peer-review process is largely unknown. This article presents the results of a two-year survey of reviewers at Monthly Weather Review and calculates the contribution of reviews by individual reviewers and collectively to the system as a whole. DATA AND METHODS.To improve the peerreview process qualitatively and quantitatively by gathering information that might decrease the time to editors' decisions for the benefit of authors, a brief survey was designed to find out more about how peer reviewers of Monthly Weather Review perform their reviews. Given that the principal impediment to timely decisions is the wait for late reviews, a decrease in the time that manuscripts reside with reviewers results in faster decisions. This survey was conducted over a 2-yr period from

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

“…Publications are related to scientists' recruitment and promotion, grant funding, invitations to speak at conferences, establishment of collaborations, and media coverage. If the time of review work done by scientists is transformed to money estimated by salary, the annual expenses may add up to more than 2 million euros per international journal (Golden & Schultz, 2012).…”

“…The current double-blind review system works in parallel with the quantitative markers of research quality (bibliometrics), such as impact factors, citations, downloads in databases, views in social media, and journal rankings (Golden & Schultz, 2012;Hirschauer, 2015;Wani & Zainab, 2017;Wellcome Trust, 2015;Wicherts, 2016). Metric practices do not directly serve authors' actual needs to write a methodologically coherent research article.…”

Peer Reviewer Non-performance: The Need to Replenish the “Commons”

“…According to Kovanis et al's (2016) analysis, an even higher number of manuscripts could be peer reviewed, which indicates that the biomedics field does not suffer from a peer review crisis. Research in other fields such as ecology (Petchey et al, 2014), meteorology (Golden, & Schultz, 2012), science (Vines, Rieseberg, & Smith, 2010), and science and technology (House of Commons Science and Technology Committee, 2011) supports this result. However, we do not know if we can generalize these results to the IS field and conclude that there IS peer review does not suffer from a crisis.…”

“…The most productive reviewers-defined as those who review more than six manuscripts in a year-completed 79 percent of all reviews and averaged almost twice as many reviews per year (i.e., 14.3) compared with eight manuscripts per year for less active reviewers (Ware & Monkman, 2008). Similarly, researchers have report referee overloading to exist in other fields (Golden & Schultz, 2012; House of Commons Science and Technology Committee, 2011; Petchey et al, 2014). Golden and Schultz (2012) report that the number of reviews completed per year, in the meteorology field, averages eight per year.…”

Peer Review of Datasets: When, Why, and How

“…Peer review pipelines are already facing an exploding number of journal article submissions and grant applications (Miller and Couzin 2007). A recent survey by Golden and Schultz (2012) of reviewers for the publication Monthly Weather Review found that reviewers already review an average of eight journal articles per year, spending on average 9.6 h per review. No comparable study has been done specifically looking at the time and effort required to perform data peer review, although anecdotal evidence suggests that in some cases the this effort might be considerably less than 9.6 h (Callaghan 2015).…”