On The Peer Review Reports: Does Size Matter?

Abstract

Scientometers and sociologists of science have spilled much ink on the topic of peer review over the past twenty years given its primordial role in a context marked by the exponential growth of scientific production and the proliferation of predatory journals. Although the topic is addressed under different prisms, few studies have empirically analyzed to what extent it can affect the quality of publications. Here we study (1) the link between the length of reviewers' reports and the citations received by publications, (2) The threshold (expressed in number of words in reviewers' reports) at which the effect on citations becomes statistically significant. To do this, we used data from the Publons database (58,093 peer review reports). We have adjusted this sample to match the WoS database structure. Our regression results show that peer review positively affects the quality of publications. In other words, the more in-depth (longer) the referees' reports are, the greater the publication improvements will be, resulting in an increase in citations received. This result is important from both the point of view of reviewers and that of journal's chiefs-editors. Even if it is not a remunerated activity, it is important that it be more valued at least within the framework of research evaluation exercises, given its positive impact on science. On the thresholds, the results differ between the types of regression used. The threshold is 1250 words (about one page) for Negative Binomial regression, 1400 words for Poisson regression and 2400 words for GLM regression. These results suggest that a report that brings the most value to the publication is a deep report, which naturally requires more time to produce. This is an important result in a context where the deadlines granted to reviewers tend to decrease for some publishers.

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

“…13,14 This premise is supported by research by Yadav, 15 who found that review volume was positively associated with overall review quality. Additionally, a recent study 16 found that lengthier reviews were positively associated with the number of citations received by the article (arguably a measure of article quality). Thus, under the assumption that a very low word count reflects, at least to some degree, the quality of a peer review, this study aimed to determine the prevalence of very short peer reviews in 3 leading general medical journals: The BMJ, PLOS Medicine, and BMC Medicine.…”

“…This premise is supported by research by Yadav, who found that review volume was positively associated with overall review quality. Additionally, a recent study found that lengthier reviews were positively associated with the number of citations received by the article (arguably a measure of article quality).…”