To explain or not to explain?—Artificial intelligence explainability in clinical decision support systems

Abstract

Explainability for artificial intelligence (AI) in medicine is a hotly debated topic. Our paper presents a review of the key arguments in favor and against explainability for AI-powered Clinical Decision Support System (CDSS) applied to a concrete use case, namely an AI-powered CDSS currently used in the emergency call setting to identify patients with life-threatening cardiac arrest. More specifically, we performed a normative analysis using socio-technical scenarios to provide a nuanced account of the role of explainability for CDSSs for the concrete use case, allowing for abstractions to a more general level. Our analysis focused on three layers: technical considerations, human factors, and the designated system role in decision-making. Our findings suggest that whether explainability can provide added value to CDSS depends on several key questions: technical feasibility, the level of validation in case of explainable algorithms, the characteristics of the context in which the system is implemented, the designated role in the decision-making process, and the key user group(s). Thus, each CDSS will require an individualized assessment of explainability needs and we provide an example of how such an assessment could look like in practice.

ePOCT+ and the medAL-suite: Development of an electronic clinical decision support algorithm and digital platform for pediatric outpatients in low- and middle-income countries

“…Understanding how a diagnosis and treatment plan is reached is fundamental to clinical and patient autonomy, important for continued learning, and for fostering trust in any algorithm. [64][65][66] Efforts were made to present simple decision tree logic for each diagnosis. Nevertheless, the optimal method of presentation of algorithm branches to assure understanding by primary care level healthcare workers should be further explored.…”

Piloting a Survey-Based Assessment of Transparency and Trustworthiness with Three Medical AI Tools

“…We assumed that each medical AI product has application boundaries [ 24 , 31 ] that should be reported and scored zero transparency and trustworthiness points if these were not disclosed. Similar to a previous study [ 35 ], it was challenging to judge if all the potential sources of bias, causes of harm, and caveats for deployment were sufficiently investigated. It was also challenging to judge whether bias mitigation steps are required or not and assign justified scores.…”

“…It was also challenging to judge whether bias mitigation steps are required or not and assign justified scores. Scoring answers on the performed validation steps (e.g., model uncertainty and feature importance) was challenging, because the methods for these validation steps have not yet been standardized and may require adaptation to individual use cases [ 32 , 35 ]. Other assessors may find it relevant to score questions on additional info on model development or validation.…”

Building Efficient CNN Architectures for Histopathology Images Analysis: A Case-Study in Tumor-Infiltrating Lymphocytes Classification

“…For instance, these methods require a significant amount of annotated data to be used in training, which is particularly costly in digital pathology as it requires an expert pathologist to manually annotate large volumes of data ( 11 , 12 ). Also, applications developed with deep learning should consider explainability to improve confidence in their use ( 13 , 14 ).…”