Artificial Intelligence Chatbots’ Understanding of the Risks and Benefits of Computed Tomography and Magnetic Resonance Imaging Scenarios

Abstract

Purpose: Patients may seek online information to better understand medical imaging procedures. The purpose of this study was to assess the accuracy of information provided by 2 popular artificial intelligence (AI) chatbots pertaining to common imaging scenarios’ risks, benefits, and alternatives. Methods: Fourteen imaging-related scenarios pertaining to computed tomography (CT) or magnetic resonance imaging (MRI) were used. Factors including the use of intravenous contrast, the presence of renal disease, and whether the patient was pregnant were included in the analysis. For each scenario, 3 prompts for outlining the (1) risks, (2) benefits, and (3) alternative imaging choices or potential implications of not using contrast were inputted into ChatGPT and Bard. A grading rubric and a 5-point Likert scale was used by 2 independent reviewers to grade responses. Prompt variability and chatbot context dependency were also assessed. Results: ChatGPT’s performance was superior to Bard’s in accurately responding to prompts per Likert grading (4.36 ± 0.63 vs 3.25 ± 1.03 seconds, P < .0001). There was substantial agreement between independent reviewer grading for ChatGPT (κ = 0.621) and Bard (κ = 0.684). Response text length was not statistically different between ChatGPT and Bard (2087 ± 256 characters vs 2162 ± 369 characters, P = .24). Response time was longer for ChatGPT (34 ± 2 vs 8 ± 1 seconds, P < .0001). Conclusions: ChatGPT performed superior to Bard at outlining risks, benefits, and alternatives to common imaging scenarios. Generally, context dependency and prompt variability did not change chatbot response content. Due to the lack of detailed scientific reasoning and inability to provide patient-specific information, both AI chatbots have limitations as a patient information resource.

Should We Embrace Our Robot Overlords?

“…In the article "Artificial Intelligence Chatbots' Understanding of the Risks and Benefits of Computed Tomography and Magnetic Resonance Imaging Scenarios," Patil et al 4 investigate the accuracy of information provided by 2 of the most popular AI LLM, or "chatbots"; ChatGPT, developed by OpenAI and Bard, developed by Google. These LLM were asked questions regarding the benefits, risks, and alternatives to CT and MRI in relation to a number of common clinical queries.…”

“…A score of 1 was "incorrect and potentially harmful" whereas a score of 5 was considered to be "complete with no errors or false claims." 4 Despite its focus on a small number of clinical scenarios and subset of imaging modalities, the study by Patil et al highlights some important considerations regarding the use of LLM by both patients and physicians.…”

“…Questions (prompts) were submitted to the chatbots from the perspective of both patients and physicians and the responses graded on a Likert scale based on the accuracy of the information returned. A score of 1 was “incorrect and potentially harmful” whereas a score of 5 was considered to be “complete with no errors or false claims.” 4 Despite its focus on a small number of clinical scenarios and subset of imaging modalities, the study by Patil et al highlights some important considerations regarding the use of LLM by both patients and physicians.…”

“…This conversational to-and-fro serves to make information all the more accessible to those who decide to engage with it. Patil et al 4 acknowledge that potentially important contextual information, such as patient characteristics and co-morbidities, were not provided in their prompts. Although highlighted as a limitation of the study, this is likely representative of how non-medical individuals might interact with the LLM, as they may be unaware of the clinical implication of these details.…”

“…Bard’s answer contained the following: “Not all CTPAs require intravenous iodinated contrast. If your doctor does not think that the benefits of intravenous contrast outweigh the risks, you may be able to have a CTPA without contrast.” 4 To the eye of any trained radiologist, this answer is obviously incorrect, as CTPA requires contrast. However, to the enquiring patient, the error may not be so apparent.…”