A Scoping Review on the Progress, Applicability, and Future of Explainable Artificial Intelligence in Medicine

Abstract

Due to the success of artificial intelligence (AI) applications in the medical field over the past decade, concerns about the explainability of these systems have increased. The reliability requirements of black-box algorithms for making decisions affecting patients pose a challenge even beyond their accuracy. Recent advances in AI increasingly emphasize the necessity of integrating explainability into these systems. While most traditional AI methods and expert systems are inherently interpretable, the recent literature has focused primarily on explainability techniques for more complex models such as deep learning. This scoping review critically analyzes the existing literature regarding the explainability and interpretability of AI methods within the clinical domain. It offers a comprehensive overview of past and current research trends with the objective of identifying limitations that hinder the advancement of Explainable Artificial Intelligence (XAI) in the field of medicine. Such constraints encompass the diverse requirements of key stakeholders, including clinicians, patients, and developers, as well as cognitive barriers to knowledge acquisition, the absence of standardised evaluation criteria, the potential for mistaking explanations for causal relationships, and the apparent trade-off between model accuracy and interpretability. Furthermore, this review discusses possible research directions aimed at surmounting these challenges. These include alternative approaches to leveraging medical expertise to enhance interpretability within clinical settings, such as data fusion techniques and interdisciplinary assessments throughout the development process, emphasizing the relevance of taking into account the needs of final users to design trustable explainability methods.

Artificial Intelligence and Its Role in Diagnosing Heart Failure: A Narrative Review

“…AI models, especially those implementing deep learning, present significant challenges due to their "blackbox" nature [190]. The term "black box" refers to the opaque nature of the internal working processes of their learning and decision-making functions [190,191].…”

“…AI models, especially those implementing deep learning, present significant challenges due to their "blackbox" nature [190]. The term "black box" refers to the opaque nature of the internal working processes of their learning and decision-making functions [190,191]. Despite the possibility of having precise knowledge about the input data, the lack of transparency in understanding how the thinking process of the AI system led to the outcome makes it impossible to fully understand and rectify any mistakes that take place [191,192].…”

“…This also affects the clinician's trust and willingness to accept the results, produced due to the lack of detailed explanations and the inability to navigate the decision-making process of the DL models. Moreover, there is a lack of insight for a clinician about the details of the statistical approximations being done due to a scarcity of technical and mathematical knowledge [190]. These problems surrounding interpretability and explainability engender ethical and legal quandaries as well.…”

“…The absence of transparency in such circumstances may entail legal ramifications as well since patients can ask to be properly informed and explained regarding the utilization of their personal data [190]. Gonzalez-Alday et al [190] mentioned the GDPR (General Data Protection Regulation) as one of the latest examples of legal regulations about this matter, wherein the European Union has started to integrate the requirement of transparency of AI where there is sensitive data in use [190].…”

“…Furthermore, the question of legal and ethical consequences also arises from the fact that there are chances of misdiagnoses [190][191][192][193]. AI may interpret various correlated factors as causation without fully understanding the reasons behind their relationship, basically overlooking important connections or mistakenly attributing factors as causation [190][191][192][193].…”