The researchers using AI to analyse peer review

Abstract

Anna Severin and her team used artificial intelligence to analyse peer-review reports. Do more-highly cited journals have higherquality peer review? Reviews are generally confidential and the definition of 'quality' is elusive, so this is a difficult question to answer. But researchers who used machine learning to study 10,000 peerreview reports in biomedical journals have tried. They invented proxy measures for quality, which they term thoroughness and helpfulness. Their work, reported in a preprint 1 in July, found that reviews at journals with higher impact factors seem to spend more time discussing a paper's methods but less time on suggesting improvements than do reviews at lowerimpact journals. However, the differences between high-and low-impact journals were modest and variability was high. The authors say this suggests that a journal's impact factor is "a bad predictor for the quality of review of an individual manuscript". Anna Severin, who led the study as part of her PhD in science policy and scholarly publishing at the University of Bern and the Swiss National Science Foundation (SNSF) in Bern, spoke to Nature about the work. Severin is now a health consultant at management consultancy Capgemini Invent in Germany.

Using GPT-4 to write a scientific review article: a pilot evaluation study

“…The interest in ChatGPT's application across scientific domains is evident. Studies have evaluated ChatGPT's potential in clinical and academic writing [3][4][5][6][7][8][9][10], and discussions are underway about its use as a scientific review article generator [11,12,13]. However, many of these studies predate the release of the more advanced GPT-4, which may render their findings outdated.…”

“…The swift progress in Natural Language Processing (NLP) technology, particularly with the rise of Generative Pre-trained Transformers (GPT) and other Large Language Models (LLMs), has equipped researchers with a potent tool for swiftly processing extensive literature. A recent survey indicates that ChatGPT has become an asset for researchers across various fields [3]. For instance, a PubMed search for “ChatGPT” yielded over 1,400 articles with ChatGPT in their titles as of November 30th, 2023, marking a significant uptake just one year after ChatGPT’s introduction.…”

Convergence analysis of artificial intelligence research capacity: Are the less developed catching up with the developed ones?

“…This highlights the importance of investing in AI research, which not only drives innovation but also stimulates economic development by creating new industries and job opportunities (Aghion et al, 2018; Caltech Science Exchange, 2023; Chubb et al, 2022; Liu et al, 2020). However, the enthusiasm for AI's integration into research and its anticipated importance in the coming decade is tempered by concerns regarding over‐reliance on pattern recognition that could bypass fundamental understanding and the potential for AI to entrench biases or discrimination (Noorden & Perkel, 2023; Sifat, 2023). These apprehensions highlight the necessity of a deliberate and ethical approach to AI development, ensuring that advancements in AI research contribute positively to society and mitigate any adverse impacts.…”

Low and high-impact-factor journals: which has better peer review quality?

“…Therefore, its quality can be used as an appropriate indicator for the journal and the quality of its publications. Therefore, one key question has long been asked in the debate: Do high-impact-factor journals have better peer review quality than low-impact-factor journals?Attempting to shed light on the relationship between the journal impact factor and quality, a group of researchers, led by Anna Severin (University of Bern), have recently examined 10,000 peer review reports submitted to 1,644 medical and life sciences journals [2,3]. They use artificial intelligence to analyze two measures proxying the peer review quality: thoroughness and helpfulness.…”

“…Attempting to shed light on the relationship between the journal impact factor and quality, a group of researchers, led by Anna Severin (University of Bern), have recently examined 10,000 peer review reports submitted to 1,644 medical and life sciences journals [2,3]. They use artificial intelligence to analyze two measures proxying the peer review quality: thoroughness and helpfulness.…”