Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods

Abstract

Artificial Intelligence (AI) has emerged as a useful aid in numerous clinical applications for diagnosis and treatment decisions. Deep neural networks have shown same or better performance than clinicians in many tasks owing to the rapid increase in the available data and computational power. In order to conform to the principles of trustworthy AI, it is essential that the AI system be transparent, robust, fair and ensure accountability. Current deep neural solutions are referred to as black-boxes due to a lack of understanding of the specifics concerning the decision making process. Therefore, there is a need to ensure interpretability of deep neural networks before they can be incorporated in the routine clinical workflow. In this narrative review, we utilized systematic keyword searches and domain expertise to identify nine different types of interpretability methods that have been used for understanding deep learning models for medical image analysis applications based on the type of generated explanations and technical similarities. Furthermore, we report the progress made towards evaluating the explanations produced by various interpretability methods. Finally we discuss limitations, provide guidelines for using interpretability methods and future directions concerning the interpretability of deep neural networks for medical imaging analysis.

A Weakly Supervised Gradient Attribution Constraint for Interpretable Classification and Anomaly Detection

“…Most attribution methods [17], [28] have been proposed as posthoc procedure to visualize which pixels contribute positively or negatively to the network decision [20], [21]. In few works, they have also been used to regularize the training of a classification network.…”

“…From a trained network, these methods compute a heat map that indicates the positive or negative contribution of each voxel of the input image in the network decision. These methods are mostly used at inference to verify the interpretability of a trained network and check that these maps match high-level knowledge as in [20], [21]. For example, in a medical context, they can be used to check that the decision of the network matches anatomical abnormalities present in the image.…”