ChatGPT passing USMLE shines a spotlight on the flaws of medical education

Mbakwe, Amarachi B.; Lourentzou, Ismini; Celi, Leo Anthony; Mechanic, Oren J.; Dagan, Alon

doi:10.1371/journal.pdig.0000205

Cited by 172 publications

(105 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the area of health care education, ChatGPT appears to have a massive transformative potential. The need to rethink and revise the current assessment tools in health care education comes in light of ChatGPT’s ability to pass reputable exams (e.g., USMLE) and possibility of ChatGPT misuse, which would result in academic dishonesty [ 24 , 34 , 58 , 59 , 62 , 76 , 85 , 86 , 87 ].…”

Section: Discussionmentioning

confidence: 99%

“…Specifically, in ophthalmology examination, Antaki et al showed that ChatGPT currently performed at the level of an average first-year resident [ 70 ]. Such a result highlights the need to focus on questions involving the assessment of critical and problem-based thinking [ 34 ]. Additionally, the utility of ChatGPT in health care education can involve tailoring education based on the needs of the student with immediate feedback [ 46 ].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

2023

View full text Add to dashboard Cite

ChatGPT is an artificial intelligence (AI)-based conversational large language model (LLM). The potential applications of LLMs in health care education, research, and practice could be promising if the associated valid concerns are proactively examined and addressed. The current systematic review aimed to investigate the utility of ChatGPT in health care education, research, and practice and to highlight its potential limitations. Using the PRIMSA guidelines, a systematic search was conducted to retrieve English records in PubMed/MEDLINE and Google Scholar (published research or preprints) that examined ChatGPT in the context of health care education, research, or practice. A total of 60 records were eligible for inclusion. Benefits of ChatGPT were cited in 51/60 (85.0%) records and included: (1) improved scientific writing and enhancing research equity and versatility; (2) utility in health care research (efficient analysis of datasets, code generation, literature reviews, saving time to focus on experimental design, and drug discovery and development); (3) benefits in health care practice (streamlining the workflow, cost saving, documentation, personalized medicine, and improved health literacy); and (4) benefits in health care education including improved personalized learning and the focus on critical thinking and problem-based learning. Concerns regarding ChatGPT use were stated in 58/60 (96.7%) records including ethical, copyright, transparency, and legal issues, the risk of bias, plagiarism, lack of originality, inaccurate content with risk of hallucination, limited knowledge, incorrect citations, cybersecurity issues, and risk of infodemics. The promising applications of ChatGPT can induce paradigm shifts in health care education, research, and practice. However, the embrace of this AI chatbot should be conducted with extreme caution considering its potential limitations. As it currently stands, ChatGPT does not qualify to be listed as an author in scientific articles unless the ICMJE/COPE guidelines are revised or amended. An initiative involving all stakeholders in health care education, research, and practice is urgently needed. This will help to set a code of ethics to guide the responsible use of ChatGPT among other LLMs in health care and academia.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

2023

View full text Add to dashboard Cite

show abstract

“…In the area of healthcare education, ChatGPT appears to have a massive transformative potential. The need to rethink and revise the current assessment tools in healthcare education comes in light of ChatGPT’s ability to pass reputable exams (e.g., USMLE), and possibility of ChatGPT misuse that would result in increased academic dishonesty [28,34,64,66,68,72].…”

Section: Discussionmentioning

confidence: 99%

“…Specifically, in ophthalmology examination, Antaki et al showed that ChatGPT currently performed at the level of an average first-year resident [51]. The focus on questions involving the assessment of critical and problem-based thinking appears of utmost value [66]. Additionally, the utility of ChatGPT in healthcare education can involve tailoring education based on the student needs with immediate feedback [32].…”

Section: Discussionmentioning

confidence: 99%

The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations

Sallam

2023

Preprint

195

125

View full text Add to dashboard Cite

An artificial intelligence (AI)-based conversational large language model (LLM) was launched in November 2022 namely, ChatGPT. Despite the wide array of potential applications of LLMs in healthcare education, research and practice, several valid concerns were raised. The current systematic review aimed to investigate the possible utility of ChatGPT and to highlight its limitations in healthcare education, research and practice. Using the PRIMSA guidelines, a systematic search was conducted to retrieve English records in PubMed/MEDLINE and Google Scholar under the term ChatGPT. Eligibility criteria included the published research or preprints of any type that discussed ChatGPT in the context of healthcare education, research and practice. A total of 280 records were identified, and following full screening, a total of 60 records were eligible for inclusion. Benefits/applications of ChatGPT were cited in 51/60 (85.0%) records with the most common being the utility in scientific writing followed by benefits in healthcare research (efficient analysis of massive datasets, code generation and rapid concise literature reviews besides utility in drug discovery and development). Benefits in healthcare practice included cost saving, documentation, personalized medicine and improved health literacy. Concerns/possible risks of ChatGPT use were expressed in 58/60 (96.7%) records with the most common being the ethical issues including the risk of bias, plagiarism, copyright issues, transparency issues, legal issues, lack of originality, incorrect responses, limited knowledge, and inaccurate citations. Despite the promising applications of ChatGPT which can result in paradigm shifts in healthcare education, research and practice, the embrace of this application should be done with extreme caution. Specific applications of ChatGPT in health education include the promising utility in personalized learning tools and shift towards more focus on critical thinking and problem-based learning. In healthcare practice, ChatGPT can be valuable for streamlining the workflow and refining personalized medicine. Saving time for the focus on experimental design and enhancing research equity and versatility are the benefits in scientific research. Regarding authorship in scientific articles, as it currently stands, ChatGPT does not qualify to be listed as an author unless the ICMJE/COPE guidelines are revised and amended. An initiative involving all stakeholders involved in healthcare education, research and practice is urgently needed to set a code of ethics and conduct on the responsible practices involving ChatGPT among other LLMs.

show abstract

“…Algorithms perform relatively well on knowledge-based tests despite the lack of domain-specific training; ChatGPT achieved ~ 66% and ~ 72% on Basic Life Support and Advanced Cardiovascular Life Support tests, respectively ( 5 ), and performed at or near the passing threshold on the United States Medical Licensing Exam ( 6 , 7 ). However, they are notoriously bad at context and nuance ( 8 ) – two things critical for safe and effective patient care, which requires the implementation of medical knowledge, concepts, and principles in real-world settings. In their analysis of the future of employment, Frey and Osborne estimate that, while the probability of administrative health care jobs automation is relatively high (eg, 91% for health information technicians), the probability of automating the jobs of physicians and surgeons is 0.42% ( 9 ).…”

Section: Can Chatgpt Replace Physicians?mentioning

confidence: 99%

Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma

Homolak

2023

Croat Med J

126

View full text Add to dashboard Cite

ChatGPT passing USMLE shines a spotlight on the flaws of medical education

Cited by 172 publications

References 1 publication

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations

Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma

Contact Info

Product

Resources

About