What do Deep Networks Like to See?

Abstract

We propose a novel way to measure and understand convolutional neural networks by quantifying the amount of input signal they let in. To do this, an autoencoder (AE) was fine-tuned on gradients from a pre-trained classifier with fixed parameters. We compared the reconstructed samples from AEs that were fine-tuned on a set of image classifiers (AlexNet, VGG16, ResNet-50, and Inception v3) and found substantial differences. The AE learns which aspects of the input space to preserve and which ones to ignore, based on the information encoded in the backpropagated gradients. Measuring the changes in accuracy when the signal of one classifier is used by a second one, a relation of total order emerges. This order depends directly on each classifier's input signal but it does not correlate with classification accuracy or network size. Further evidence of this phenomenon is provided by measuring the normalized mutual information between original images and auto-encoded reconstructions from different fine-tuned AEs. These findings break new ground in the area of neural network understanding, opening a new way to reason, debug, and interpret their results. We present four concrete examples in the literature where observations can now be explained in terms of the input signal that a model uses.

TSInsight: A Local-Global Attribution Framework for Interpretability in Time Series Data

“…Finally, we compute the attributions from the trained auto-encoder (step 3) followed by the sanity check using our suppression test (step 4). We will first cover some basic background and then dive into the formulation of the problem presented by Palacio et al [11]. We will then present the proposed formulation, adapting the basic one for the interpretability of deep learning-based time series models.…”

“…The first stream for explainable systems, which attempts to explain pretrained models using attribution techniques, has been a major focus of research in the few past years. The most common strategy is to visualize the filters of the deep model [11,[13][14][15][16]. This is very effective for visual modalities since images are directly intelligible for humans.…”

“…Palacio et al [11] introduced yet another approach to understand a deep model by leveraging auto-encoders. After training both the classifier and the auto-encoder in isolation, they attached the auto-encoder to the head of the classifier and fine-tuned only the decoder, freezing the parameters of the classifier and the encoder.…”

“…Rather than optimizing the auto-encoder to reconstruct the whole input, we optimize the network to only reconstruct parts which are useful for the classifier, i.e., are correlated or causal for the prediction. Although auto-encoders have been used to understand the classifier in the past [11], we extend this to the task of attribution, where relevant features of the input have to be highlighted. For this purpose, it is important for the reconstructions to be aligned with the inputs.…”

DeepLSF: Fusing Knowledge and Data for Time Series Forecasting

“…al. [44], where pre-training an Encoder-Decoder system at the input layer produced more robust results which were less prone to adversarial attacks in image classification tasks.…”