Factors Influencing Clinician Trust in Predictive Clinical Decision Support Systems for In-Hospital Deterioration: Qualitative Descriptive Study

Abstract

Background Clinician trust in machine learning–based clinical decision support systems (CDSSs) for predicting in-hospital deterioration (a type of predictive CDSS) is essential for adoption. Evidence shows that clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy. Objective The aim of this study was to explore the phenomenon of clinician trust in predictive CDSSs for in-hospital deterioration by confirming and characterizing factors known to influence trust (understandability and accuracy), uncovering and describing other influencing factors, and comparing nurses’ and prescribing providers’ trust in predictive CDSSs. Methods We followed a qualitative descriptive methodology conducting directed deductive and inductive content analysis of interview data. Directed deductive analyses were guided by the human-computer trust conceptual framework. Semistructured interviews were conducted with nurses and prescribing providers (physicians, physician assistants, or nurse practitioners) working with a predictive CDSS at 2 hospitals in Mass General Brigham. Results A total of 17 clinicians were interviewed. Concepts from the human-computer trust conceptual framework—perceived understandability and perceived technical competence (ie, perceived accuracy)—were found to influence clinician trust in predictive CDSSs for in-hospital deterioration. The concordance between clinicians’ impressions of patients’ clinical status and system predictions influenced clinicians’ perceptions of system accuracy. Understandability was influenced by system explanations, both global and local, as well as training. In total, 3 additional themes emerged from the inductive analysis. The first, perceived actionability, captured the variation in clinicians’ desires for predictive CDSSs to recommend a discrete action. The second, evidence, described the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust. The final theme, equitability, described fairness in system predictions. The findings were largely similar between nurses and prescribing providers. Conclusions Although there is a perceived trade-off between machine learning–based CDSS accuracy and understandability, our findings confirm that both are important for fostering clinician trust in predictive CDSSs for in-hospital deterioration. We found that reliance on the predictive CDSS in the clinical workflow may influence clinicians’ requirements for trust. Future research should explore the impact of reliance, the optimal explanation design for enhancing understandability, and the role of perceived actionability in driving trust.

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

“…In settings ranging from primary care clinics to hospital wards, explanations for ML predictions have been necessary to aid physician understanding and build trust. [22][23][24] Anesthesiology clinicians using a postoperative hypotension prediction system felt that different decision thresholds were needed for different patient groups, 8 similar to our finding that different thresholds should be used in different surgical populations. However, some user needs depend on the specific use case.…”

“…A consistent message was that clinicians were more interested in identifying modifiable risk factors than nonmodifiable risk factors, consistent with prior literature. 24,26,27 Most participants wanted to use this information to drive intraoperative risk mitigation strategies. Although such approaches have previously been used for ML applications in anesthesiology and critical care, 10,11,28 caution is necessary because ML models identify associations, not causal relationships, between input and output variables.…”

Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department

“…For an application to the medical field, researchers should prepare ML-based CDSS for clinically actionable explanations. The physicians did not trust the predictions when the logic behind those was unclear 17 , 39 – 42 . We attempted to increase the explainability to physicians by presenting features that were highly influential in predicting the occurrence of the four critical events in our study and found that the features of the influence of each event were different.…”

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

“…Focussing on sociotechnical dimensions, Schwartz et al's [63] application of the human-computer trust conceptual framework to explore clinician trust is particularly noteworthy. Here, nurses and prescribers from 24 acute and intensive care units in two hospitals were interviewed about their trust in the predictive AI.…”