Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference

Abstract

State-of-the-art convolutional neural networks (CNNs) yield record-breaking predictive performance, yet at the cost of high-energy-consumption inference, that prohibits their widely deployments in resource-constrained Internet of Things (IoT) applications. We propose a dual dynamic inference (DDI) framework that highlights the following aspects: 1) we integrate both input-dependent and resource-dependent dynamic inference mechanisms under a unified framework in order to fit the varying IoT resource requirements in practice. DDI is able to both constantly suppress unnecessary costs for easy samples, and to halt inference for all samples to meet hard resource constraints enforced; 2) we propose a flexible multigrained learning to skip (MGL2S) approach for input-dependent inference which allows simultaneous layer-wise and channelwise skipping; 3) we extend DDI to complex CNN backbones such as DenseNet and show that DDI can be applied towards optimizing any specific resource goals including inference latency or energy cost. Extensive experiments demonstrate the superior inference accuracy-resource trade-off achieved by DDI, as well as the flexibility to control such trade-offs compared to existing peer methods. Specifically, DDI can achieve up to 4 times computational savings with the same or even higher accuracy as compared to existing competitive baselines.

Overview of the Neural Network Compression and Representation (NNR) Standard

“…One issue that makes creating compact NNs for efficient inference challenging is the dependency on the characteristics of the target hardware. A framework for dynamic inference on resource constrained hardware, including input-and resource dependent dynamic inference mechanisms, allowing to meet specific resource constraints, has been proposed recently [25]. First steps are being made towards network compression methods that output representations prepared for later specialization to the target platform [26].…”

Scenario Based Run-Time Switching for Adaptive CNN-Based Applications at the Edge

“…As a second case, when the application is designed using data-driven adaptive methodologies, such as [12,14,25,31,34], the CNN execution is sensitive to the input data complexity. To process "easy" images, they may use a lower resolution or fewer layers, whereas processing "hard" images requires more computation.…”

“…This may lead to increased UAV power consumption over the flight duration and, eventually, to the violation of the application power constraint, causing an emergency landing as illustrated in Figure 2(h). Thus, the methodologies in [12,14,25,31,34] are not suitable for CNN-based applications executed at the edge in changing environment, because these can neither properly adapt the application to the environment variations, nor guarantee that the application constantly meets platform-aware constraints.…”

Adaptive Inference through Early-Exit Networks

“…FlexDNN [12] Vision/Classification Footprint overhead-aware design of EE-networks. DDI [76] Vision/Classification Combines layer/channel skipping with early exiting. MESS [40] Vision/Segmentation Image-level EE based on difficulty for semantic segmentation.…”