Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks

Nam, Woo Hyun; Gur, Shir; Choi, Jaesik; Wolf, Lior; Lee, Seong–Whan

doi:10.48550/arxiv.1904.00605

Cited by 8 publications

(15 citation statements)

References 23 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Relative Attributing Propagation RAP [18] decomposes the output predictions of DNNs in terms of relative influence among the neurons, resulting in assigning the relevant and irrelevant attributions with a bi-polar importance. By changing the perspective from value to influence, the generated visual explanations show the characteristics of strong objectness and clear distinction of relevant and irrelevant attributions.…”

Section: Layerwise Relevance Propagationmentioning

confidence: 99%

“…DeepLIFT [22] propagates the differences of contribution scores between the activated neurons and their reference activation. Recently proposed Relative attributing propagation (RAP) [18] is a method for decomposing the positive (relevant) and negative (irrelevant) relevance to each neuron, according to its relative influence among the neurons. By changing the perspective from value to influence, it shows a clear distinction of relevant/irrelevant attributions with a high objectness score.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units

Nam,

Lee

2021

Preprint

Self Cite

View full text Add to dashboard Cite

As interpretability has been pointed out as the obstacle to the adoption of Deep Neural Networks (DNNs), there is an increasing interest in solving a transparency issue to guarantee the impressive performance. In this paper, we demonstrate the efficiency of recent attribution techniques to explain the diagnostic decision by visualizing the significant factors in the input image. By utilizing the characteristics of objectness that DNNs have learned, fully decomposing the network prediction visualizes clear localization of target lesion. To verify our work, we conduct our experiments on Chest X-ray diagnosis with publicly accessible datasets. As an intuitive assessment metric for explanations, we report the performance of intersection of Union between visual explanation and bounding box of lesions. Experiment results show that recently proposed attribution methods visualize the more accurate localization for the diagnostic decision compared to the traditionally used CAM. Furthermore, we analyze the inconsistency of intentions between humans and DNNs, which is easily obscured by high performance. By visualizing the relevant factors, it is possible to confirm that the criterion for decision is in line with the learning strategy. Our analysis of unmasking machine intelligence represents the necessity of explainability in the medical diagnostic decision.

show abstract

Section: Layerwise Relevance Propagationmentioning

confidence: 99%

mentioning

confidence: 99%

Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units

Nam,

Lee

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The task of generating a heatmap that indicates local relevancy from the perspective of a CNN observing an input image has been tackled from many different directions, including gradient-based methods [32,33,31], attribution methods [2,24,25,11], and image manipulation methods [6,7,22].…”

Section: Explainabilitymentioning

confidence: 99%

Explainability Guided Multi-Site COVID-19 CT Classification

Ameen¹,

Shaharabany²,

Wolf³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Radiologist examination of chest CT is an effective way for screening COVID-19 cases. In this work, we overcome three challenges in the automation of this process: (i) the limited number of supervised positive cases, (ii) the lack of region-based supervision, and (iii) the variability across acquisition sites. These challenges are met by incorporating a recent augmentation solution called SnapMix, by a new patch embedding technique, and by performing a test-time stability analysis. The three techniques are complementary and are all based on utilizing the heatmaps produced by the Class Activation Mapping (CAM) explainability method. Compared to the current state of the art, we obtain an increase of five percent in the F1 score on a site with a relatively high number of cases, and a gap twice as large for a site with much fewer training images.

show abstract

“…Pointing Game Pointing Game (Zhang et al 2018) assesses the attribution methods by computing the matching Method mIOU Grad-CAM (threshold: mean) + CRF 52.14 Segmentation Prop (Guillaumin, Küttel, and Ferrari 2014a) 57.30 DeepMask (Pinheiro, Collobert, and Dollár 2015) 58.69 RAP (Nam et al 2019) 59.46 DeepSaliency (Li et al 2016) 62.12 Pixel Objectness (Xiong, Jain, and Grauman 2018) 64.22 RSP 60.81 RSP + CRF 64.51…”

Section: Evaluating Quality Of Attributionsmentioning

confidence: 99%

Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations

Nam¹,

Choi

Lee³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

The clear transparency of Deep Neural Networks (DNNs) is hampered by complex internal structures and nonlinear transformations along deep hierarchies. In this paper, we propose a new attribution method, Relative Sectional Propagation (RSP), for fully decomposing the output predictions with the characteristics of class-discriminative attributions and clear objectness. We carefully revisit some shortcomings of backpropagation-based attribution methods, which are trade-off relations in decomposing DNNs. We define hostile factor as an element that interferes with finding the attributions of the target and propagate it in a distinguishable way to overcome the non-suppressed nature of activated neurons. As a result, it is possible to assign the bi-polar relevance scores of the target (positive) and hostile (negative) attributions while maintaining each attribution aligned with the importance. We also present the purging techniques to prevent the decrement of the gap between the relevance scores of the target and hostile attributions during backward propagation by eliminating the conflicting units to channel attribution map. Therefore, our method makes it possible to decompose the predictions of DNNs with clearer class-discriminativeness and detailed elucidations of activation neurons compared to the conventional attribution methods. In a verified experimental environment, we report the results of the assessments: (i) Pointing Game, (ii) mIoU, and (iii) Model Sensitivity with PAS-CAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing backward decomposition methods, including distinctive and intuitive visualizations.

show abstract

Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks

Cited by 8 publications

References 23 publications

Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units

Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units

Explainability Guided Multi-Site COVID-19 CT Classification

Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations

Contact Info

Product

Resources

About