Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology

Agarwal, Monika; Sharma, Priyanka; Goswami, Ayan

doi:10.7759/cureus.40977

Cited by 26 publications

(22 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a cross-sectional study, Agarwal et.al [22] However, the difficulty level was lower compared to Bard and Bing (Table 2).…”

Section: Descriptive Summary Of Resultsmentioning

confidence: 88%

See 1 more Smart Citation

Large language models for generating medical examinations: systematic review

Artsi,

Sorin,

Konen

et al. 2024

Preprint

View full text Add to dashboard Cite

PurposeWriting multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs.MethodsThe authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. MEDLINE was used as a search database.ResultsOverall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans.All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify.ConclusionsLLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations.

show abstract

“…In a cross-sectional study, Agarwal et.al [22] However, the difficulty level was lower compared to Bard and Bing (Table 2).…”

Section: Descriptive Summary Of Resultsmentioning

confidence: 88%

“…In a cross-sectional study, Agarwal et.al [22] compared different LLMs. They compared Chat-GPT 3.5/Bard/Bing in MCQs generating capability.…”

Section: Resultsmentioning

confidence: 99%

Large language models for generating medical examinations: systematic review

Artsi,

Sorin,

Konen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these are just suppositions as we did not compare our method with others, such as AIG-using chatbots or zero-shot MCQs. Previous studies attained scoring inferences of chatbots developed MCQs without using learning objectives (Agarwal et al, 2023;Ayub et al, 2023). However, these studies did not assess for generalization or extrapolation inferences.…”

Section: Discussionmentioning

confidence: 99%

“…A crosssectional study used ChatGPT, Google Bard and Microsoft Bing to develop MCQs for a physiology course, in this study a careful blueprint was mapped by two content experts; inferences of scoring and generalization were collected. However, generalization was collected subjectively by asking content experts to classify the di culty of MCQs (Agarwal et al, 2023). Other study used a qualitative methodology to assess MCQs developed by ChatGPT and ChatPDF for the dermatology board examination.…”

Section: Literature Reviewmentioning

confidence: 99%

Using chatbots to develop multiple-choice questions. We got evidence, but we ain't there yet!

Flores-Cohaila,

Calderón,

Castro-Blancas

et al. 2023

Preprint

View full text Add to dashboard Cite

Developing accessible assessment tools is crucial for educators. Traditional methods demand significant resources such as time and expertise. Therefore, an accessible, user-friendly approach is needed. Traditional assessment creation faces challenges, however, new solutions like automatic item generation have emerged. Despite their potential, they still require expert knowledge. ChatGPT and similar chatbots offer a novel approach in this field. Our study evaluates the validity of MCQs generated by chatbots under the Kane validity framework. We focused on the top ten topics in Infectious and Tropical diseases, chosen based on epidemiological data and expert evaluations. These topics were transformed into learning objectives for chatbots like GPT-4, BingAI, and Claude to generate MCQs. Each chatbot produced 10 MCQs, which were subsequently refined. We compared 30 chatbot-generated MCQs with 10 from a Peruvian medical examination. The participants included 48 medical students and doctors from Peru. Our analysis revealed that the quality of chatbot-generated MCQs is consistent with those created by humans. This was evident in scoring inferences, with no significant differences in difficulty and discrimination indexes. In conclusion, chatbots appear to be a viable tool for creating MCQs in the field of infectious and tropical diseases in Peru. Although our study confirms their validity, further research is necessary to optimize their use in educational assessments.

show abstract

“…Recently, numerous articles have emerged about AI applications in different fields of medicine such as radiology, dermatology, physiology, hematology, ophthalmology, biochemistry, parasitology, neurosurgery, forensic medicine, dental education, etc. [5][6][7][8][9][10][11][12][13][14]. These models have been helpful in solving complex medical problems, interpreting radiology reports, being used to diagnose diseases, writing scientific articles or answering and generating different medical exam questions and have shown varying degrees of accuracy in these fields [2][3][4][5]7,8,10,11,15].…”

Section: Introductionmentioning

confidence: 99%

The Significance of Artificial Intelligence Platforms in Anatomy Education: An Experience With ChatGPT and Google Bard

Ilgaz,

Çelik

2023

Cureus

View full text Add to dashboard Cite

This study evaluated the use of two large language models (LLMs), ChatGPT and Google Bard, in anatomy education. The models were asked to answer questions, generate multiple-choice questions, and write articles on anatomy topics. The results showed that the models were able to perform these tasks with varying degrees of accuracy. ChatGPT and Google Bard did not differ significantly in terms of answering questions. Both models were able to generate multiple-choice questions with a high degree of accuracy. However, the performance of the models in article writing was not yet at a sufficient level. The study also found that the use of LLMs in medical education requires caution. This is because LLMs are still under development and they can sometimes generate inaccurate or misleading information. It is important to carefully evaluate the output of LLMs before using them in educational settings. Overall, the study found that LLMs have the potential to be valuable tools for anatomy education. However, more research is needed to improve the accuracy of the models and to better understand how they can be used effectively in educational settings.

show abstract

Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology

Cited by 26 publications

References 25 publications

Large language models for generating medical examinations: systematic review

Large language models for generating medical examinations: systematic review

Using chatbots to develop multiple-choice questions. We got evidence, but we ain't there yet!

The Significance of Artificial Intelligence Platforms in Anatomy Education: An Experience With ChatGPT and Google Bard

Contact Info

Product

Resources

About