Understanding the peer review endeavor in scientific publishing

Zhang, Guangyao; Xu, Shenmeng; Sun, Yao; Chun-lin, Jiang; Wang, Xianwen

doi:10.1016/j.joi.2022.101264

Cited by 15 publications

(4 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, our study is based on the premise that whether or not a review is very short holds some valuable information in reflecting review quality. 13,14 This premise is supported by research by Yadav, 15 who found that review volume was positively associated with overall review quality. Additionally, a recent study 16 found that lengthier reviews were positively associated with the number of citations received by the article (arguably a measure of article quality).…”

Section: Introductionmentioning

confidence: 79%

“…Conversely, a review that is long may not necessarily be a high-quality review if it lacks substantive content. Nevertheless, our study is based on the premise that whether or not a review is very short holds some valuable information in reflecting review quality . This premise is supported by research by Yadav, who found that review volume was positively associated with overall review quality.…”

Section: Introductionmentioning

confidence: 82%

See 1 more Smart Citation

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

Geldsetzer,

Heemann,

Tikka

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceHigh-quality peer reviews are often thought to be essential to ensuring the integrity of the scientific publication process, but measuring peer review quality is challenging. Although imperfect, review word count could potentially serve as a simple, objective metric of review quality.ObjectiveTo determine the prevalence of very short peer reviews and how often they inform editorial decisions on research articles in 3 leading general medical journals.Design, Setting, and ParticipantsThis cross-sectional study compiled a data set of peer reviews from published, full-length original research articles from 3 general medical journals (The BMJ, PLOS Medicine, and BMC Medicine) between 2003 and 2022. Eligible articles were those with peer review data; all peer reviews used to make the first editorial decision (ie, accept vs revise and resubmit) were included.Main Outcomes and MeasuresPrevalence of very short reviews was the primary outcome, which was defined as a review of fewer than 200 words. In secondary analyses, thresholds of fewer than 100 words and fewer than 300 words were used. Results were disaggregated by journal and year. The proportion of articles for which the first editorial decision was made based on a set of peer reviews in which very short reviews constituted 100%, 50% or more, 33% or more, and 20% or more of the reviews was calculated.ResultsIn this sample of 11 466 reviews (including 6086 in BMC Medicine, 3816 in The BMJ, and 1564 in PLOS Medicine) corresponding to 4038 published articles, the median (IQR) word count per review was 425 (253-575) words, and the mean (SD) word count was 520.0 (401.0) words. The overall prevalence of very short (&lt;200 words) peer reviews was 1958 of 11 466 reviews (17.1%). Across the 3 journals, 843 of 4038 initial editorial decisions (20.9%) were based on review sets containing 50% or more very short reviews. The prevalence of very short reviews and share of editorial decisions based on review sets containing 50% or more very short reviews was highest for BMC Medicine (693 of 2585 editorial decisions [26.8%]) and lowest for The BMJ (76 of 1040 editorial decisions [7.3%]).Conclusion and RelevanceIn this study of 3 leading general medical journals, one-fifth of initial editorial decisions for published articles were likely based at least partially on reviews of such short length that they were unlikely to be of high quality. Future research could determine whether monitoring peer review length improves the quality of peer reviews and which interventions, such as incentives and norm-based interventions, may elicit more detailed reviews.

show abstract

Section: Introductionmentioning

confidence: 79%

Section: Introductionmentioning

confidence: 82%

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

Geldsetzer,

Heemann,

Tikka

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

show abstract

“…Participation at Publons was also highly skewed, in terms of disciplines, and with some journals or publishers being more comprehensively covered by Publons than others, most likely as a result of agreements between those journals and/or publishers and Clarivate [ 39 ], with some reviewers being highly—perhaps excessively—productive [ 21 , 40 ], and the majority being male [ 41 ]. Rice et al [ 42 ] confirmed this latter finding by examining the peer reports of Publons-registered “mega peer reviewers”, i.e., reviewers that had reviewed at least 100 papers in a 1-year period (in 2018), finding that 74% of them were male, compared to 58% of “standard” peer reviewers, i.e., those who had reviewed 18 or fewer papers per year.…”

Section: Publons’ Role In Peer Review Rewards: a Critical Perspectivementioning

confidence: 99%

The Role of Publons in the Context of Open Peer Review

Silva¹,

Nazarovets

2022

Pub Res Q

View full text Add to dashboard Cite

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

show abstract

“… Mhaidli & Schaub (2021) adopted the scenario construction method to identify the use of manipulative techniques in advertising. Such classification and identification methods are mostly based on subjective experience, which is difficult to adapt to the personalized scene needs of large-scale user groups ( Van Helvert & Fowler, 2003 ; Zhang & Wang, 2022 ). In this article, we attempt to establish a user scenario identification model by using BERT, Skip-gram model and other data mining methods to extract usage scenarios from online reviews.…”

Section: Introductionmentioning

confidence: 99%

Scenario-feature identification from online reviews based on BERT

Xun-jiang¹,

Kang²

2023

PeerJ Computer Science

View full text Add to dashboard Cite

Scenario endows a product with meanings. It has become the key to win the competition to design a product according to specific usage scene. Traditional scenario identification and product feature association methods have disadvantages such as subjectivity, high cost, coarse granularity, and limited scenario can be identified. In this regard, we propose a BERT-based scenario-feature identification model to effectively extract the information about users’ experience and usage scene from online reviews. First, the scenario-feature identification framework is proposed to depict the whole identification process. Then, the BERT-based scene-sentence recognition model is constructed. The Skip-gram and word vector similarity methods are used to construct the scene and feature lexicon. Finally, the triad is constructed through the analysis of scene-feature co-occurrence matrix, which realizes the association of scenario and product features. This proposed model is of great practical value for product developers to better understand customer’s requirements in specific scenarios. The experiments of scenario-feature identification from the reviews of Pacific Auto verifies the effectiveness of this method.

show abstract

Understanding the peer review endeavor in scientific publishing

Cited by 15 publications

References 66 publications

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

The Role of Publons in the Context of Open Peer Review

Scenario-feature identification from online reviews based on BERT

Contact Info

Product

Resources

About