Application of ChatGPT in multilingual medical education: How does ChatGPT fare in 2023's Iranian residency entrance examination

Khorshidi, Hamid; Mohammadi, Afshin; Yousem, David M.; Abolghasemi, Jamileh; Ansari, Golnoosh; Mirza-Aghazadeh-Attari, Mohammad; Acharya, U Rajendra; Abbasian Ardakani, Ali

doi:10.1016/j.imu.2023.101314

Cited by 14 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ChatGPT's performance has been acceptable not only in English-based medical exams but also in tests conducted in other languages; for instance, the Japanese medical licensing examination, 16 the Chinese National Medical Licensing Examination (NMLE) 17 and the Iranian Medical Residency Examination. 18 The Certification Examination in Family Medicine conducted by the College of Family Physicians of Canada (CFPC) is a comprehensive assessment of broad clinical knowledge in the field of family medicine in Canada. 19 This exam consists of the oral component, the simulated office oral exam, and the written component, consisting of short-answer management problems (SAMPs).…”

Section: How This Study Might Affect Research Practice or Policymentioning

confidence: 99%

Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada

Mousavi,

Shafiee,

Harley

et al. 2024

Fam Med Com Health

View full text Add to dashboard Cite

IntroductionThe application of large language models such as generative pre-trained transformers (GPTs) has been promising in medical education, and its performance has been tested for different medical exams. This study aims to assess the performance of GPTs in responding to a set of sample questions of short-answer management problems (SAMPs) from the certification exam of the College of Family Physicians of Canada (CFPC).MethodBetween August 8th and 25th, 2023, we used GPT-3.5 and GPT-4 in five rounds to answer a sample of 77 SAMPs questions from the CFPC website. Two independent certified family physician reviewers scored AI-generated responses twice: first, according to the CFPC answer key (ie, CFPC score), and second, based on their knowledge and other references (ie, Reviews’ score). An ordinal logistic generalised estimating equations (GEE) model was applied to analyse repeated measures across the five rounds.ResultAccording to the CFPC answer key, 607 (73.6%) lines of answers by GPT-3.5 and 691 (81%) by GPT-4 were deemed accurate. Reviewer’s scoring suggested that about 84% of the lines of answers provided by GPT-3.5 and 93% of GPT-4 were correct. The GEE analysis confirmed that over five rounds, the likelihood of achieving a higher CFPC Score Percentage for GPT-4 was 2.31 times more than GPT-3.5 (OR: 2.31; 95% CI: 1.53 to 3.47; p<0.001). Similarly, the Reviewers’ Score percentage for responses provided by GPT-4 over 5 rounds were 2.23 times more likely to exceed those of GPT-3.5 (OR: 2.23; 95% CI: 1.22 to 4.06; p=0.009). Running the GPTs after a one week interval, regeneration of the prompt or using or not using the prompt did not significantly change the CFPC score percentage.ConclusionIn our study, we used GPT-3.5 and GPT-4 to answer complex, open-ended sample questions of the CFPC exam and showed that more than 70% of the answers were accurate, and GPT-4 outperformed GPT-3.5 in responding to the questions. Large language models such as GPTs seem promising for assisting candidates of the CFPC exam by providing potential answers. However, their use for family medicine education and exam preparation needs further studies.

show abstract

Section: How This Study Might Affect Research Practice or Policymentioning

confidence: 99%

Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada

Mousavi,

Shafiee,

Harley

et al. 2024

Fam Med Com Health

View full text Add to dashboard Cite

show abstract

“…Harnessing GPT-4 for generation of cybersecurity GRC policies: A focus on ransomware attack mitigation [273]. Application of ChatGPT in multilingual medical education: How does ChatGPT fare in 2023's Iranian residency entrance examination [263]. Research on the impact of trends related to ChatGPT [300].…”

Section: Fig 5 Ai Components For Nlpmentioning

confidence: 99%

“…They're testing it to see if it can understand and respond like a human and also making sure it's used responsibly and ethically [241], [249], [258], [263], [265], [268], [269], [273], [284], [286], [287], [300], [301], [303] T5 (Text-to-Text Transfer Transformer)…”

Section: Fig 5 Ai Components For Nlpmentioning

confidence: 99%

Evolving Conversations: A Review of Chatbots and Implications in Natural Language Processing for Cultural Heritage Ecosystems

Suryanto,

Wibawa,

Hariyono

et al. 2023

IJRCS

View full text Add to dashboard Cite

Chatbot technology, a rapidly growing field, uses Natural Language Processing (NLP) methodologies to create conversational AI bots. Contextual understanding is essential for chatbots to provide meaningful interactions. Still, to date chatbots often struggle to accurately interpret user input due to the complexity of natural language and diverse fields, hence the need for a Systematic Literature Review (SLR) to investigate the motivation behind the creation of chatbots, their development procedures and methods, notable achievements, challenges and emerging trends. Through the application of the PRISMA method, this paper contributes to revealing the rapid and dynamic progress in chatbot technology with NLP learning models, enabling sophisticated and human-like interactions on the trends observed in chatbots over the past decade. The results, from various fields such as healthcare, organization and business, virtual personalities, to education, do not rule out the possibility of being developed in other fields such as chatbots for cultural preservation while suggesting the need for supervision in the aspects of language comprehension bias and ethics of chatbot users. In the end, the insights gained from SLR have the potential to contribute significantly to the advancement of chatbots on NLP as a comprehensive field.

show abstract

“…In the context of medical education, where precision and depth of understanding can have crucial implications, evaluating AI’s efficacy and relationship to relevant outcomes is essential. 6 , 7 Moreover, comparing the performance of AI models like ChatGPT to that of medical students may provide insights into areas where AI can complement human learning and, importantly, where it might fall short 8 , 9 Several studies have investigated the capabilities of AI in medical education, exploring both its potential and limitations. 10–13 Analyzing MCQs in medical education is crucial as it allows educators to assess the effectiveness of questions in testing higher-order thinking and clinical reasoning skills, ensuring assessments accurately reflect the skills required for medical practice.…”

Section: Introductionmentioning

confidence: 99%

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

Bharatha,

Ojeh,

Fazle Rabbi

et al. 2024

AMEP

View full text Add to dashboard Cite

Introduction This research investigated the capabilities of ChatGPT-4 compared to medical students in answering MCQs using the revised Bloom’s Taxonomy as a benchmark. Methods A cross-sectional study was conducted at The University of the West Indies, Barbados. ChatGPT-4 and medical students were assessed on MCQs from various medical courses using computer-based testing. Results The study included 304 MCQs. Students demonstrated good knowledge, with 78% correctly answering at least 90% of the questions. However, ChatGPT-4 achieved a higher overall score (73.7%) compared to students (66.7%). Course type significantly affected ChatGPT-4’s performance, but revised Bloom’s Taxonomy levels did not. A detailed association check between program levels and Bloom’s taxonomy levels for correct answers by ChatGPT-4 showed a highly significant correlation (p<0.001), reflecting a concentration of “remember-level” questions in preclinical and “evaluate-level” questions in clinical courses. Discussion The study highlights ChatGPT-4’s proficiency in standardized tests but indicates limitations in clinical reasoning and practical skills. This performance discrepancy suggests that the effectiveness of artificial intelligence (AI) varies based on course content. Conclusion While ChatGPT-4 shows promise as an educational tool, its role should be supplementary, with strategic integration into medical education to leverage its strengths and address limitations. Further research is needed to explore AI’s impact on medical education and student performance across educational levels and courses.

show abstract

Application of ChatGPT in multilingual medical education: How does ChatGPT fare in 2023's Iranian residency entrance examination

Cited by 14 publications

References 13 publications

Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada

Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada

Evolving Conversations: A Review of Chatbots and Implications in Natural Language Processing for Cultural Heritage Ecosystems

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

Contact Info

Product

Resources

About