Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare

Abstract

Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in "black box" models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the "user." Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders needs, ensuring better implementation and operation in a healthcare context.

Towards greater neuroimaging classification transparency via the integration of explainability methods and confidence estimation approaches

“…While explainability is needed for the development of clinical decision support systems [ 11 , 37 – 39 ], a number of researchers have indicated that current explainability approaches are insufficient for use in a clinical setting [ 38 , 61 ]. There are valid concerns associated with their critiques.…”

“…However, in and of themselves, they are insufficient to the task. It is also critical that automated neuroimaging-based clinical decision support systems be explainable [ 11 , 37 – 39 ]. If clinicians are to use clinical decision support systems, they are ethically obligated to be able to explain the recommendations of such systems to their patients [ 11 ].…”

Identifying Neuropsychiatric Disorder Subtypes and Subtype-dependent Variation in Diagnostic Deep Learning Classifier Performance

“…If neuroimaging clinical decision support systems (CDSS) are ever to be implemented in a clinical setting, they must be both robust and reliable [1]. One aspect of this reliability is that clinicians and need to not only know whether there are systematic differences in how the model will perform for different patients [2].…”

A rapid review of machine learning approaches for telemedicine in the scope of COVID-19

“…Besides the use of AI to enhance experiences of both clinicians and patients [33] in healthcare. Furthermore, the growth of AI explainable learning has made the AI understandable, providing explanations catered to humans, promoting transparency about the decisions and consequently reliability, [37] , [45] . Once the healthcare professionals and patients do not accept decisions without understanding and trusting the explanations, or at least how the decisions are made [46] .…”