Evaluating Large Language Models for the National Premedical Exam in India: Comparative Analysis of GPT-3.5, GPT-4, and Bard (Preprint)

Abstract

BACKGROUND Large language models (LLMs) have revolutionized Natural Language Processing (NLP) with their ability to generate human-like text through extensive training on large datasets. These models, including ChatGPT-3.5, GPT-4, and Bard, find applications beyond NLP, attracting interest from academia and industry. Students are actively leveraging LLMs to enhance learning experiences and prepare for high-stakes exams, such as the National Eligibility cum Entrance Test (NEET) in India. OBJECTIVE This comparative analysis aims to evaluate the performance of ChatGPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. METHODS In this paper, we test the performance of ChatGPT 3.5, GPT-4, and Bard on pre-medical exam in India, NEET-2023. The questions of NEET were provided to these AI models, and the responses were recorded. Precision, recall, accuracy and F1 score were used to evaluate the performance of all three models. RESULTS GPT-4 demonstrated consistent superiority over Bard and ChatGPT-3.5 in all three subjects. Specifically, GPT-4 achieved accuracy rates of 72.5% in Physics, 44.44% in Chemistry, and 50.5% in Biology. CONCLUSIONS The study's findings provide valuable insights into the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. GPT-4 emerged as the most accurate model, highlighting its potential for educational applications. The results underscore the suitability of LLMs for high-stakes exams and their positive impact on education. Additionally, the study establishes a benchmark for evaluating and enhancing LLMs' performance in educational tasks, promoting responsible and informed use of these models in diverse learning environments.

Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment

“…On medical school entrance examinations, ChatGPT has performed accurately on the National Eligibility cum Entrance Test (NEET) in India (41). ChatGPT-4 scored 72.5%, 44.4%, and 50.5% in physics, chemistry, and biology on the NEET, respectively.…”

To Compare the Efficiency of ChatGPT and Bard in Medical Education: An Analysis of MCQ-Based Learning and Assessment

“…Google Bard, like its counterpart ChatGPT, demonstrates remarkable proficiency in swiftly retrieving, interpreting and delivering information. Its versatility is evident, as it can not only offer information on a specific topic but also generate content tailored to specific requirements [5][6] .…”

ChatGPT's Applications in Higher Education: Unmasking Opportunities and Challenges

“…In response to these concerns, OpenAI has created ChatGPT-3.5, a language model with a parameter count of 6.7 billion (Ray, 2023). The revised iteration of the system exhibited a decrease in biases, an augmentation in safety and ethical considerations, and an enhancement in its capacity to manage various types of communication (Farhat, Chaudry, Nadeem, Sohail, & Madsen, 2023;Ray, 2023). The unveiling of ChatGPT-4, the latest iteration of the AI chatbot, occurred in March 2023.…”