ECQ$^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs

Abstract

The remarkable success of deep neural networks (DNNs) in various applications is accompanied by a significant increase in network parameters and arithmetic operations. Such increases in memory and computational demands make deep learning prohibitive for resourceconstrained hardware platforms such as mobile devices. Recent efforts aim to reduce these overheads, while preserving model performance as much as possible, and include parameter reduction techniques, parameter quantization, and lossless compression techniques.In this chapter, we develop and describe a novel quantization paradigm for DNNs: Our method leverages concepts of explainable AI (XAI) and concepts of information theory: Instead of assigning weight values based on their distances to the quantization clusters, the assignment function additionally considers weight relevances obtained from Layer-wise Relevance Propagation (LRP) and the information content of the clusters (entropy optimization). The ultimate goal is to preserve the most relevant weights in quantization clusters of highest information content. Experimental results show that this novel Entropy-Constrained and XAI-adjusted Quantization (ECQ x ) method generates ultra low-precision (2-5 bit) and simultaneously sparse neural networks while maintaining or even improving model performance. Due to reduced parameter precision and high number of zero-elements, the rendered networks are highly compressible in terms of file size, up to 103× compared to the full-precision unquantized DNN model. Our approach was evaluated on different types of models and datasets (including Google Speech Commands and CIFAR-10) and compared with previous work.

From "Where" to "What": Towards Human-Understandable Explanations through Concept Relevance Propagation

“…Prominent examples are DeepLIFT [122], Deconvolution [147], LRP [11] and Deep Taylor Decomposition [83]. Notably, LRP has been shown to achieve faithful intermediate attribution as can be seen in [18,144]. These methods have the advantage that they efficiently provide attributions for intermediate neurons "for free" as a by-product, without any additional algorithmic extensions of the aforementioned.…”

“…These methods have the advantage that they efficiently provide attributions for intermediate neurons "for free" as a by-product, without any additional algorithmic extensions of the aforementioned. However, this by-product has usually been ignored in the literature except for a few works that use this information to directly improve specific aspects of deep models [81,144,128,18], or regard them as proxy representations of explanations for the identification and eradication of systematic Clever Hans behavior [6].…”

“…Layer-wise Relevance Propagation, like other backpropagation-based methods (e.g., [147,126,122]), computes attribution scores for all hidden units of a neural network model in order to allot a score to the model input. While in some recent works those hidden layer attribution scores have been used as a (not further semantically interpreted) means to improve deep models [81,144,128,18], or as proxy representations of explanations for the identification of systematic Clever Hans behavior [6], they are usually disregarded as a "by-product" for obtaining per-sample explanations in input space. The reason is fairly simple: End-to-end learned representations of data in latent space are usually difficult (or impossible) to interpret, other than the samples in input space, e.g., images.…”

“…With those practical limitations in mind, we want to address the question of how much explanation is needed in order to sufficiently understand a particular inference of the model. Recent work on network compression and pruning has shown that contemporary neural network architectures often are highly over-parameterized and a large amount of filters can be removed without noticeable impact on the performance of the model [146,144,18]. The explicit representation of prototypes in some neural network architectures [27] has been exploited to remove spurious information from the decision making in a user-interactive fashion [42].…”