The performance of <scp>ChatGPT</scp> in generating answers to clinical questions in psychiatry: a two‐layer assessment

Luykx, Jurjen J.; Gerritse, Frank; Habets, Philippe C.; Vinkers, Christiaan H.

doi:10.1002/wps.21145

Cited by 20 publications

(7 citation statements)

References 6 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another similar use of ChatGPT was as an aid to answer clinical questions. A recent study evaluated the performance of users (psychiatrist and medical residents in the Netherlands) using ChatGPT as compared to nonusers for answering several questions in psychiatry, and it was observed that the users had better and faster responses as compared to nonusers [ 35 ]. Although these applications differ from this study, they might hint that ChatGPT currently has a database that holds relevant data in the field of psychiatry, which might explain the realism of scenarios and prompts observed for SCTs 2, 3, and 4.…”

Section: Discussionmentioning

confidence: 99%

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Hudon,

Kiepura,

Pelletier

et al. 2024

JMIR Med Educ

View full text Add to dashboard Cite

Background Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)–based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials. Objective The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion. Methods This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs. Results A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content. Conclusions This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

show abstract

Section: Discussionmentioning

confidence: 99%

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Hudon,

Kiepura,

Pelletier

et al. 2024

JMIR Med Educ

View full text Add to dashboard Cite

show abstract

“…Research indicates that ChatGPT excels in accuracy, completeness, nuance, and speed when generating responses to clinical inquiries in psychiatry [22]. Moreover, LLMs like ChatGPT play a pivotal role in automating the evaluation of medical literature, facilitating the identification of accurately reported research findings [23].…”

Section: Large Language Models In Medical Researchmentioning

confidence: 99%

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Luo,

Chen,

Zhu

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

Large language models (LLMs) like ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be employed to expedite various steps, including defining clinical questions, literature search, document screening, information extraction, and language refinement, etc, thereby conserving resources and enhancing efficiency. However, when utilizing LLMs, attention should be given to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. This article reviews the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors involved in systematic reviews and meta-analyses.

show abstract

“…OpenAI is not transparent on how GPT-4 was trained, so it is unclear whether scientific research, often behind paywalls, was included in the vast data sets that were integrated into its network during training [3]. For example, a study into ChatGPT's knowledge of clinical psychiatry found evidence for its promising accuracy, completeness, nuance, and speed, but also revealed a lack of pharmaceutical knowledge, which is typically found in textbooks rather than the web-based information ChatGPT was trained on [4].…”

Section: Potential Harmsmentioning

confidence: 99%

Can we use ChatGPT for Mental Health and Substance Use Education? Examining Its Quality and Potential Harms

Spallek,

Birrell,

Kershaw

et al. 2023

JMIR Med Educ

View full text Add to dashboard Cite

Background The use of generative artificial intelligence, more specifically large language models (LLMs), is proliferating, and as such, it is vital to consider both the value and potential harms of its use in medical education. Their efficiency in a variety of writing styles makes LLMs, such as ChatGPT, attractive for tailoring educational materials. However, this technology can feature biases and misinformation, which can be particularly harmful in medical education settings, such as mental health and substance use education. This viewpoint investigates if ChatGPT is sufficient for 2 common health education functions in the field of mental health and substance use: (1) answering users’ direct queries and (2) aiding in the development of quality consumer educational health materials. Objective This viewpoint includes a case study to provide insight into the accessibility, biases, and quality of ChatGPT’s query responses and educational health materials. We aim to provide guidance for the general public and health educators wishing to utilize LLMs. Methods We collected real world queries from 2 large-scale mental health and substance use portals and engineered a variety of prompts to use on GPT-4 Pro with the Bing BETA internet browsing plug-in. The outputs were evaluated with tools from the Sydney Health Literacy Lab to determine the accessibility, the adherence to Mindframe communication guidelines to identify biases, and author assessments on quality, including tailoring to audiences, duty of care disclaimers, and evidence-based internet references. Results GPT-4’s outputs had good face validity, but upon detailed analysis were substandard in comparison to expert-developed materials. Without engineered prompting, the reading level, adherence to communication guidelines, and use of evidence-based websites were poor. Therefore, all outputs still required cautious human editing and oversight. Conclusions GPT-4 is currently not reliable enough for direct-consumer queries, but educators and researchers can use it for creating educational materials with caution. Materials created with LLMs should disclose the use of generative artificial intelligence and be evaluated on their efficacy with the target audience.

show abstract

The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two‐layer assessment

Cited by 20 publications

References 6 publications

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Can we use ChatGPT for Mental Health and Substance Use Education? Examining Its Quality and Potential Harms

Contact Info

Product

Resources

About