ChatGPT could be the reviewer of your next scientific paper. Evidence on the limits of AI-assisted academic reviews

Abstract

The irruption of artificial intelligence (AI) in all areas of our lives is a reality to which the university, as an institution of higher education, must respond prudently, but also with no hesitation. This paper discusses the potential that resources based on AI presents as potential reviewers of scientific articles in a hypothetical peer review of already published articles. Using different models (GPT-3.5 and GPT-4) and platforms (ChatPDF and Bing), we obtained three full reviews, both qualitative and quantitative, for each of the five articles examined, thus being able to delineate and contrast the results of all of them in terms of the human reviews that these same articles received at the time. The evidence found highlights the extent to which we can and should rely on generative language models to support our decisions as qualified experts in our field. Furthermore, the results also corroborate the hallucinations inherent in these models while pointing out one of their current major shortcomings: the context window limit. On the other hand, the study also points out the inherent benefits of a model that is in a clear expansion phase, providing a detailed view of the potential and limitations that these models offer as possible assistants to the review of scientific articles, a key process in the communication and dissemination of academic research.

Capítulo 3. Inteligencia Artificial en la comunicación científica

“…Este ejemplo demuestra cómo los modelos de lenguaje basados en inteligencia artificial pueden desempeñar un papel en el análisis de datos y la escritura científica. Carabantes et al (2023) realizaron un estudio en el que se emplearon diversos modelos de lenguaje generativos, como GPT-3.5 y GPT-4, junto con plataformas como ChatPDF y Bing, para obtener revisiones completas y compararlas con revisiones humanas previas de cinco artículos publicados. Los resultados mostraron una gran capacidad y versatilidad al momento de evaluar los artículos, pero también se presentaron serias limitaciones como las constantes alucinaciones y el límite de la ventana contextual.…”