Analyzing sentiments in peer review reports: Evidence from two science funding agencies

Luo, Junwen; Feliciani, Thomas; Reinhart, Martin; Hartstein, Judith; Das, V. Tulasi; Alabi, Olalere; Shankar, K.

doi:10.1162/qss_a_00156

Cited by 15 publications

(9 citation statements)

References 38 publications

(56 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reviewers with more neutral sentiment scored higher on these two metrics. It has been demonstrated that review sentiment is a strong indicator for predicting proposals’ funding decisions (Luo et al , 2021). Our finding extends this conclusion to review quality.…”

Section: Discussionmentioning

confidence: 99%

Assessing and predicting the quality of peer reviews: a text mining approach

Meng

2023

View full text Add to dashboard Cite

Purpose This paper aims to quantify the quality of peer reviews, evaluate them from different perspectives and develop a model to predict the review quality. In addition, this paper investigates effective features to distinguish the reviews' quality. Design/methodology/approach First, a fine-grained data set including peer review data, citations and review conformity scores was constructed. Second, metrics were proposed to evaluate the quality of peer reviews from three aspects. Third, five categories of features were proposed in terms of reviews, submissions and responses using natural language processing (NLP) techniques. Finally, different machine learning models were applied to predict the review quality, and feature analysis was performed to understand effective features. Findings The analysis results revealed that reviewers become more conservative and the review quality becomes worse over time in terms of these indicators. Among the three models, random forest model achieves the best performance on all three tasks. Sentiment polarity, review length, response length and readability are important factors that distinguish peer reviews’ quality, which can help meta-reviewers value more worthy reviews when making final decisions. Originality/value This study provides a new perspective for assessing review quality. Another originality of the research lies in the proposal of a novelty task that predict review quality. To address this task, a novel model was proposed which incorporated various of feature sets, thereby deepening the understanding of peer reviews.

show abstract

Section: Discussionmentioning

confidence: 99%

Assessing and predicting the quality of peer reviews: a text mining approach

Meng

2023

View full text Add to dashboard Cite

show abstract

“…Based on a dataset of scientific peer reviews from PeerRead , a deep learning network has been used to predict the acceptance or rejection of articles from peer review reports and to generate the final meta‐review, finding ‘good consistency between the recommended decisions and original decisions’, with 74%–86% accuracy at predicting the binary decision accept/reject, which was better than standard machine learning algorithms and prior bespoke peer review judgement algorithms (Pradhan et al, 2021, p. 237). There is also evidence that sentiment analysis of review reports could be helpful to predict the final decision (acceptance or rejection) of conference papers (Chakraborty et al, 2020; Ghosal, Verma, et al, 2019; Wang & Wan, 2018) or review scores of funding programmes (Luo et al, 2021). For example, PeerJudge uses AI‐assisted sentiment detection to estimate the strength of praise and criticism in peer review reports on academic papers that could be useful for editorial management decisions based on analysing a large number or review reports.…”

Section: Making Review Decisions From Peer Review Commentsmentioning

confidence: 99%

Artificial intelligence to support publishing and peer review: A summary and review

Kousha

Thelwall

2023

Learned Publishing

View full text Add to dashboard Cite

Technology is being developed to support the peer review processes of journals, conferences, funders, universities, and national research evaluations. This literature and software summary discusses the partial or complete automation of several publishing‐related tasks: suggesting appropriate journals for an article, providing quality control for submitted papers, finding suitable reviewers for submitted papers or grant proposals, reviewing, and review evaluation. It also discusses attempts to estimate article quality from peer review text and scores as well as from post‐publication scores but not from bibliometric data. The literature and existing examples of working technology show that automation is useful for helping to find reviewers and there is good evidence that it can sometimes help with initial quality control of submitted manuscripts. Much other software supporting publishing and editorial work exists and is being used, but without published academic evaluations of its efficacy. The value of artificial intelligence (AI) to support reviewing has not been clearly demonstrated yet, however. Finally, whilst peer review text and scores can theoretically have value for post‐publication research assessment, it is not yet widely enough available to be a practical evidence source for systematic automation.

show abstract

“…The results showed that the addition of sentiment information in review comments significantly improves the prediction ability of the system. Luo et al (2021) discussed whether review sentiment could be used as a reliable proxy for understanding peer reviewer opinions based on the combination of methods and data sets from two national funding agency contexts. The review sentiments on both specific review subjects and proposals' overall funding worthiness measured reviewer opinions.…”

Section: Content Mining Of Peer Reviewmentioning

confidence: 99%

Characterizing Peer Review Comments of Academic Articles in Multiple Rounds

Ruxue

Haomin

Jiangtao

et al. 2022

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Peer review can evaluate the quality of academic articles involving the evaluation of some aspects, e.g., methodology, experiment, and aspects are usually the critical sections, substance and properties of the article concerned by reviewers. Previous research on content mining of peer review did not distinguish the round of reviews. Detecting differences peer review comments among different rounds can help us to understand the focus change of the reviewers in the different rounds. This paper takes the Nature Communications as an example to build the corpora of peer reviews in multiple rounds. We correlate review rounds and citations and extract aspects from the corpora to analyze the round characteristics of review comments. We find that there is no significant correlation between the review rounds and the citations frequency, but when the number of rounds is within 1–3, there is a weak positive correlation between the review rounds and citations. Additionally, the reviewers tend to pay more attention to the results analysis and the significance of the work firstly, and then they will focus on the details of the article, such as diagrams. This study can provide new ideas for peer review mining and the application of bibliometrics.

show abstract

Analyzing sentiments in peer review reports: Evidence from two science funding agencies

Cited by 15 publications

References 38 publications

Assessing and predicting the quality of peer reviews: a text mining approach

Assessing and predicting the quality of peer reviews: a text mining approach

Artificial intelligence to support publishing and peer review: A summary and review

Characterizing Peer Review Comments of Academic Articles in Multiple Rounds

Contact Info

Product

Resources

About