Evaluating Large Language Models for the National Premedical Exam in India: Comparative Analysis of GPT-3.5, GPT-4, and Bard

Abstract

Background Large language models (LLMs) have revolutionized natural language processing with their ability to generate human-like text through extensive training on large data sets. These models, including Generative Pre-trained Transformers (GPT)-3.5 (OpenAI), GPT-4 (OpenAI), and Bard (Google LLC), find applications beyond natural language processing, attracting interest from academia and industry. Students are actively leveraging LLMs to enhance learning experiences and prepare for high-stakes exams, such as the National Eligibility cum Entrance Test (NEET) in India. Objective This comparative analysis aims to evaluate the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. Methods In this paper, we evaluated the performance of the 3 mainstream LLMs, namely GPT-3.5, GPT-4, and Google Bard, in answering questions related to the NEET-2023 exam. The questions of the NEET were provided to these artificial intelligence models, and the responses were recorded and compared against the correct answers from the official answer key. Consensus was used to evaluate the performance of all 3 models. Results It was evident that GPT-4 passed the entrance test with flying colors (300/700, 42.9%), showcasing exceptional performance. On the other hand, GPT-3.5 managed to meet the qualifying criteria, but with a substantially lower score (145/700, 20.7%). However, Bard (115/700, 16.4%) failed to meet the qualifying criteria and did not pass the test. GPT-4 demonstrated consistent superiority over Bard and GPT-3.5 in all 3 subjects. Specifically, GPT-4 achieved accuracy rates of 73% (29/40) in physics, 44% (16/36) in chemistry, and 51% (50/99) in biology. Conversely, GPT-3.5 attained an accuracy rate of 45% (18/40) in physics, 33% (13/26) in chemistry, and 34% (34/99) in biology. The accuracy consensus metric showed that the matching responses between GPT-4 and Bard, as well as GPT-4 and GPT-3.5, had higher incidences of being correct, at 0.56 and 0.57, respectively, compared to the matching responses between Bard and GPT-3.5, which stood at 0.42. When all 3 models were considered together, their matching responses reached the highest accuracy consensus of 0.59. Conclusions The study’s findings provide valuable insights into the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. GPT-4 emerged as the most accurate model, highlighting its potential for educational applications. Cross-checking responses across models may result in confusion as the compared models (as duos or a trio) tend to agree on only a little over half of the correct responses. Using GPT-4 as one of the compared models will result in higher accuracy consensus. The results underscore the suitability of LLMs for high-stakes exams and their positive impact on education. Additionally, the study establishes a benchmark for evaluating and enhancing LLMs’ performance in educational tasks, promoting responsible and informed use of these models in diverse learning environments.

Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

“…This may be because the Internal Medicine of TCM questions were mostly presented in case formats, where LLMs showed greater pro ciency in extracting and processing internal information. Subject-level accuracy illustrates the suitability of various models for distinct domains, providing invaluable guidance for users seeking to choose a model tailored to their speci c requirements 24 .…”

“…ChatGPT, developed by OpenAI, has undergone multiple version updates since its release. The latest iteration, GPT-4, was launched on March 14, 2023, integrating enhanced language generation capabilities with improved multi-round dialogue processing 24 .…”

Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

“…The importance and implications of AI tools are swiftly increasing in various sections of society ( 34 ). In this context, this study’s findings are vital for understanding the ChatGPT knowledge for its implementation in educational settings ( 38 ).…”

“…Similarly, Farhat et al (34) provided valuable insights into the performance of ChatGPT-3.5, ChatGPT-4, and Bard in answering the questions. The authors reported that CHAT GPT-4 is the perfect model, highlighting its potential role in education.…”

Assessing the Responses of Large Language Models (ChatGPT-4, Gemini, and Microsoft Copilot) to Frequently Asked Questions in Breast Imaging: A Study on Readability and Accuracy

“…Large language models (LLMs), such as ChatGPT-4, Gemini, and Microsoft Copilot, have revolutionized the field of artificial intelligence (AI) by demonstrating an unprecedented ability to understand and generate human-like text [1][2][3]. These chatbot models are trained on diverse internet datasets, allowing them to acquire vast amounts of knowledge and language nuances [4,5]. LLMs perform a variety of tasks, from answering queries to generating coherent and contextually appropriate responses, making them potent tools for information dissemination and decision support across multiple domains [6,7].…”