Interpretation and visualization techniques for deep learning models in medical imaging

Abstract

Deep learning approaches to medical image analysis tasks have recently become popular; however, they suffer from a lack of human interpretability critical for both increasing understanding of the methods’ operation and enabling clinical translation. This review summarizes currently available methods for performing image model interpretation and critically evaluates published uses of these methods for medical imaging applications. We divide model interpretation in two categories: (1) understanding model structure and function and (2) understanding model output. Understanding model structure and function summarizes ways to inspect the learned features of the model and how those features act on an image. We discuss techniques for reducing the dimensionality of high-dimensional data and cover autoencoders, both of which can also be leveraged for model interpretation. Understanding model output covers attribution-based methods, such as saliency maps and class activation maps, which produce heatmaps describing the importance of different parts of an image to the model prediction. We describe the mathematics behind these methods, give examples of their use in medical imaging, and compare them against one another. We summarize several published toolkits for model interpretation specific to medical imaging applications, cover limitations of current model interpretation methods, provide recommendations for deep learning practitioners looking to incorporate model interpretation into their task, and offer general discussion on the importance of model interpretation in medical imaging contexts.

Fruit quality and defect image classification with conditional GAN data augmentation

“…With this in mind, regardless of the training accuracy scores attained, further analysis is needed to explain why decisions are made and predictions are given. We analyse several synthetic images through Gradient-weighted Class Activation Mapping (Grad-CAM) [23,48]. Class activation maps are produced by the convolutional neural network trained only on real images when given synthetic data as input.…”

Benchmarking Geometric Deep Learning for Cortical Segmentation and Neurodevelopmental Phenotype Prediction

“…For visualisation and interpretation, saliency maps were generated by the process of occlusion (Zeiler and Fergus, 2014;Huff et al, 2021). Specifically, hexagonal patches of the input cortical metrics were first replaced with uniform values; then these occluded images were passed through the trained models.…”

Machine learning applications in radiation oncology

“…including more variables than lung V20, will need methods for interpreting the predictions to overcome the perception of models as a 'black box'. Interpretability has been an issue for complex machine learning models especially with the use of deep learning networks for image classification [61,62], however there are techniques to visualise the focal points of models allowing users to review the factors involved in decision making [63]. Increased data availability and variation is expected to improve the development of outcome models and help to assess both changes in practice and outcome.…”