SkipNet: Learning Dynamic Routing in Convolutional Networks

Abstract

While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient. We exploit this observation by learning to skip convolutional layers on a per-input basis. We introduce SkipNet, a modified residual network, that uses a gating network to selectively skip convolutional blocks based on the activations of the previous layer. We formulate the dynamic skipping problem in the context of sequential decision making and propose a hybrid learning algorithm that combines supervised learning and reinforcement learning to address the challenges of non-differentiable skipping decisions. We show SkipNet reduces computation by 30 90% while preserving the accuracy of the original model on four benchmark datasets and outperforms the state-of-the-art dynamic networks and static compression methods. We also qualitatively evaluate the gating policy to reveal a relationship between image scale and saliency and the number of layers skipped.

Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference

“…Dynamic Layer Skipping. Many dynamic inference methods [10], [34], [35] propose to selectively execute subsets of layers in the network conditioned on each input, framed as sequential decision making. Most of them used gating networks to skip within chain-like, ResNet-style models [36].…”

“…Most of them used gating networks to skip within chain-like, ResNet-style models [36]. SkipNet [10] introduced a hybrid learning algorithm which combines supervised learning with reinforcement learning to learn the layer-wise skipping policy based on the input, enabling greater computational savings and supporting deep architectures. BlockDrop [34] trained one global policy network to skip residual blocks.…”

“…Input-Adaptive Dynamic Inference 1) MGL2S for chain-like backbones (e.g., ResNet): IADI will selectively execute a subset of inference computation based on the input complexity. A baseline for IADI would be the dynamic layer skipping method as described in [10] that learns to skip a layer or not. In comparison, enabling finergrained options, such as skipping a channel or filter, would be more flexible and potentially yield higher computational and energy efficiency.…”

“…The skipping policies are jointly learned by compact supervised gating networks (rather than as two sequential steps) together with the base network. Comparing to merely channel-wise skipping, one of the advantages of combining it with layerwise skipping is that the efforts to compute the channel-wise routing policy can be saved if that layer is skipped first, where the computational overhead of a channel gating function is 12.5% comparing to the backbone networks, while that of a layer gating function is less than 1% [10].…”

A Novel Design of Adaptive and Hierarchical Convolutional Neural Networks using Partial Reconfiguration on FPGA

“…Here, we evaluate the partial reconfiguration approach. Also, we compare three selection methods: 1) our proposed feedback procedure; 2) SkipNet method [38]; and 3) an entropy-based method [4].…”