Clinical decision‐making in benzodiazepine deprescribing by healthcare providers vs. AI‐assisted approach

Abstract

AimThe aim of this study was to compare the clinical decision‐making for benzodiazepine deprescribing between a healthcare provider (HCP) and an artificial intelligence (AI) chatbot GPT4‐ (ChatGPT‐4).MethodsWe analysed real‐world data from a Croatian cohort of community‐dwelling benzodiazepine patients (n=154) within the EuroAgeism H2020 ESR 7 project. HCPs evaluated the data using pre‐established deprescribing criteria to assess benzodiazepine discontinuation potential. The research team devised and tested AI prompts to ensure consistency with HCP judgments. An independent researcher employed ChatGPT‐4 with predetermined prompts to simulate clinical decisions for each patient case. Data derived from human‐HCP and ChatGPT‐4 decisions were compared for agreement rates and Cohen’s kappa.ResultsBoth HPC and ChatGPT identified patients for benzodiazepine deprescribing (96.1% and 89.6%, respectively), showing an agreement rate of 95% (κ=0.200, p=0.012). Agreement on four deprescribing criteria ranged from 74.7% to 91.3% (lack of indication κ= 0.352, p<0.001; prolonged use κ=0.088, p=0.280; safety concerns κ=0.123, p=0.006; incorrect dosage κ=0.264, p=0.001). Important limitations of GPT‐4 responses were identified, including 22.1% ambiguous outputs, generic answers, and inaccuracies, posing inappropriate decision‐making risks.ConclusionWhile AI‐HCP agreement is substantial, sole AI reliance poses a risk for unsuitable clinical decision‐making. This study's findings reveal both strengths and areas for enhancement of ChatGPT‐4 in the deprescribing recommendations within a real‐world sample. Our study underscores the need for additional research on chatbot functionality in patient therapy decision‐making, further fostering the advancement of AI for optimal performance.

Evaluating Accuracy and Reproducibility of Large Language Model Performance in Pharmacy Education

“…A concise task instruction (e.g., "Please help me add the last two numbers in an array together and return the result.") or a few related examples (e.g., "For an array [2,5,6,8], the result is 14.") are often sufficient for humans to successfully complete the task to a satisfactory degree.…”

“…1–3 4 Within the field of clinical pharmacy, the performance of LLMs have been tested for deprescribing benzodiazepines, identifying drug-herb interactions, and performance on a national pharmacist examination, showing early promise. 5–8 Each year it is estimated over 6.3 billion prescription medications are dispensed and over 7 million patients will experience a medication error. Given the complexity of medication data and ability of LLMs to process large datasets, they may serve as an important tool towards making medication use safer.…”

Integrating AI in Lipedema Management: Assessing the Efficacy of GPT-4 as a Consultation Assistant

“…The remarkable functionalities of LLMs have spurred an increase in research within the medical sector, exploring their potential applications. Despite the impressive performances showcased in these studies, there is a consensus among many researchers that AI is not yet fully equipped for clinical deployment [32][33][34]. A significant aspect, often underutilized or minimally employed in these studies, is prompt engineering-an element crucial for enhancing LLM performance.…”

Assessing the potential of ChatGPT-4 to accurately identify drug-drug interactions and provide clinical pharmacotherapy recommendations

“…1–4 Within the field of clinical pharmacy, the performance of LLMs have been partially explored by testing for deprescribing benzodiazepines, identifying drug-herb interactions, and performance on a national pharmacist examination, showing promise. 5–8…”