2022
DOI: 10.1371/journal.pone.0259238
|View full text |Cite
|
Sign up to set email alerts
|

Peer review analyze: A novel benchmark resource for computational analysis of peer reviews

Abstract: Peer Review is at the heart of scholarly communications and the cornerstone of scientific publishing. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary, a flawed process at the heart of science, leading to researchers arguing with its reliability and quality. These problems could also be due to the lack of studies with the peer-review texts for various proprietary and confidentiality clauses. Peer review texts could serve as a rich source of Natural Language Proces… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1
1
1

Citation Types

0
5
0

Year Published

2022
2022
2023
2023

Publication Types

Select...
7
2

Relationship

1
8

Authors

Journals

citations
Cited by 24 publications
(12 citation statements)
references
References 46 publications
0
5
0
Order By: Relevance
“…They allocated sentences to none, one, or several of 8 content categories. We selected the 8 categories based on prior work, including the Review Quality Instrument and other scales and checklists [38], and previous studies using text analysis or machine learning to assess student and peer review reports [39][40][41][42][43]. In the manual coding process, the categories were refined, taking into account the ease of operationalising categories and their intercoder reliability.…”
Section: Data Sourcesmentioning
confidence: 99%
“…They allocated sentences to none, one, or several of 8 content categories. We selected the 8 categories based on prior work, including the Review Quality Instrument and other scales and checklists [38], and previous studies using text analysis or machine learning to assess student and peer review reports [39][40][41][42][43]. In the manual coding process, the categories were refined, taking into account the ease of operationalising categories and their intercoder reliability.…”
Section: Data Sourcesmentioning
confidence: 99%
“…To improve this type of research, removing obstacles against data sharing from publishers to the community and increasing interdisciplinary, multi-approach studies combining qualitative and quantitative research is needed [ 24 ]. Not only would this help us assess the developmental role of peer review more systematically, but also this type of research could inform guidelines and arrangements to improve the fairness of peer review [ 28 , 29 ] and improve our understanding of the multiple functions and dimensions of this complex social institution called peer review [ 4 , 50 ].…”
Section: Discussionmentioning
confidence: 99%
“…Here, we assumed that the change of statistical content of manuscripts throughout the peer review process as proxied by text revisions may reveal a joint attention effort by reviewers and authors on the methodological content of manuscripts, which is one of the most important functions of peer review. As suggested by recent research, exploring the text of manuscript and peer review reports quantitatively is key to understand the scholarly communication landscape and reconstruct the complex, indirect, collaborative relationship between authors and reviewers, which typically occurs behind the confidentiality of the journal editorial process [ 29 ].…”
Section: Introductionmentioning
confidence: 99%
“…Additionally, online peer review systems expedite the vetting process, ensuring that published research upholds rigorous standards. These platforms, often integrated with machine learning algorithms, can match manuscripts with relevant experts, streamline feedback loops, and maintain transparency in the review process (Dwivedi et al, 2023;Ghosal et al, 2022;Wicherts, 2016).…”
Section: Collaborative Platforms and Peer Reviewmentioning
confidence: 99%