Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference

Abstract

While increasingly deep networks are still in general desired for achieving state-of-the-art performance, for many specific inputs a simpler network might already suffice. Existing works exploited this observation by learning to skip convolutional layers in an input-dependent manner. However, we argue their binary decision scheme, i.e., either fully executing or completely bypassing one layer for a specific input, can be enhanced by introducing finer-grained, softer decisions. We therefore propose a Dynamic Fractional Skipping (DFS) framework. The core idea of DFS is to hypothesize layer-wise quantization (to different bitwidths) as intermediate "soft" choices to be made between fully utilizing and skipping a layer. For each input, DFS dynamically assigns a bitwidth to both weights and activations of each layer, where fully executing and skipping could be viewed as two "extremes" (i.e., full bitwidth and zero bitwidth). In this way, DFS can fractionally exploit a layers expressive power during input-adaptive inference, enabling finer-grained accuracy-computational cost trade-offs. It presents a unified view to link input-adaptive layer skipping and input-adaptive hybrid quantization. Extensive experimental results demonstrate the superior tradeoff between computational cost and model expressive power (accuracy) achieved by DFS. More visualizations also indicate a smooth and consistent transition in the DFS behaviors, especially the learned choices between layer skipping and different quantizations when the total computational budgets vary, validating our hypothesis that layer quantization could be viewed as intermediate variants of layer skipping. Our source code and supplementary material are available at https://github.com/Torment123/DFS.

Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators

“…A systematic approach to find the correct precision for each layer has been shown in (Wang et al, 2019;Dong et al, 2019;Cai et al, 2020). Dynamic multi-granularity for tensors is also considered as a way of computation saving (Shen et al, 2020). Several quantization schemes have been proposed for training (Wu et al, 2018b;Banner et al, 2018;Das et al, 2018;De Sa et al, 2018;Park et al, 2018).…”

“…Therefore, mixed-precision DNN accelerators that support versatility in data types are crucial and sometimes mandatory to exploit the benefit of different software optimizations (e.g., low-bit quantization). Moreover, supporting versatility in data types can be leveraged to trade off accuracy for efficiency based on the available resources (Shen et al, 2020). Typically, mixedprecision accelerators are designed based on low precision arithmetic units, and higher precision operation can be supported by fusing the low precision arithmetic units temporally or spatially.…”

FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training

“…Inspired by [37] and following [30], we calculate the computational cost of DNNs using the effective number of MACs, i.e., (# of M ACs) * Bit a /32 * Bit b /32 for a dot product between a and b, where Bit a and Bit b denote the precision of a and b, respectively. As such, this metric is proportional to the total number of bit operations.…”

“…Dynamic/efficient DNN training. More recently dynamic inference [23,9,24,25,26,27,28,29] was developed to reduce the average inference cost, which was then extended to the most fine-grained bit level [30,31]. While energy-efficient training is more complicated than and different from inference, many insights of the latter can be lent to the former.…”