Residual Networks Behave Like Boosting Algorithms

Abstract

We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural network architectural changes with the addition of a shrinkage parameter in the identity skip-connections and using residual modules with max-norm bounds. Through this relation between ResNet and Online Boosting, novel feature representation boosting algorithms can be constructed based on altering residual modules. We demonstrate this through proposing decision tree residual modules to construct a new boosted decision tree algorithm and demonstrating generalization error bounds for both approaches; relaxing constraints within BoostResNet algorithm to allow it to be trained in an out-of-core manner. We evaluate convolution ResNet with and without shrinkage modifications to demonstrate its efficacy, and demonstrate that our online boosted decision tree algorithm is comparable to state-of-the-art offline boosted decision tree algorithms without the drawback of offline approaches.Taking inspiration from Online Boosting, we also modify the architecture of ResNet with an additional learnable shrinkage parameter (vanilla ResNet can be interpreted as Online Boosting algorithm where the shrinkage factor is fixed/unlearnable and set to 1). As this approach only modifies the neural network architecture, the same underlying ResNet algorithms can still be used.Experimentally, we compare vanilla ResNet with our modified ResNet using convolutional neural network residual network (ResNet-CNN) on multiple image datasets. Our modified ResNet shows some improvement over vanilla ResNet architecture.We also compare our boosted decision tree neural decision tree residual network on multiple benchmark datasets and their results against other decision tree ensemble methods, including Deep Neural Decision Forests [6], neural decision trees ensembled via AdaNet [7], and off-the-shelf algorithms (gradient boosting decision tree/random forest) using LightGBM [8]. In our experiments, neural decision tree residual network showed superior performance to neural decision tree variants, and comparable performance to offline tradition gradient boosting decision tree models. Related WorksIn recent years researchers have sought to understand why ResNet perform the way that they do. The BoostResNet algorithm reinterprets ResNet as a multi-channel telescoping sum boosting problem for the purpose of introducing a new algorithm for sequential training [4], providing theoretical justification for the representational power of ResNet under linear neural network constraints [3]. One interpretation of residual networks is as a collection of many paths of differing lengths which behave like a shallow ensemble; empirical studies demonstrate that residual networks introduce short paths which can carry gradients throughout the extent of very deep n...

Ensemble deep learning: A review

“…Literature shows that the boosting concept is the backbone behind well-known architectures like Deep Residual networks (He et al, 2016;Siu, 2019), AdaNet (Cortes et al, 2017) . The theoretical background for the success of the Deep Residual networks (DeepResNet) (He et al, 2016) was explained in the context of boosting theory (Huang et al, 2018).…”

GRRLN: Gated Recurrent Residual Learning Networks for code clone detection

“…As shown in Figure 4, the residual connection is a simple shortcut connection structure that connects the inputs and outputs of GRU layer. Existing studies show that residual networks behave like boosting algorithms, where the main idea is to combines different weaker learners into a stronger learner 30 . We argue that the residual connection in GRRLN would help to boost the statement feature extraction ability of the basic GRU networks.…”

“…Existing studies show that residual networks behave like boosting algorithms, where the main idea is to combines different weaker learners into a stronger learner. 30 We argue that the residual connection in GRRLN would help to boost the statement feature extraction ability of the basic GRU networks. Intuitively, the residual connection will retain most features in the data, thus forcing the GRU model to focus more on the different features of source code.…”

Ensemble deep learning: A review

“…To make BIER more robust, Hierarchical Boosted deep metric learning [69] Literature shows that the boosting concept is the backbone behind well-known architectures like Deep Residual networks [13,72], AdaNet [73] . The theoretical background for the success of the Deep Residual networks (DeepResNet) [13] was explained in the context of boosting theory [74].…”