Understanding Decoupled and Early Weight Decay

Abstract

Weight decay (WD) is a traditional regularization technique in deep learning, but despite its ubiquity, its behavior is still an area of active research. Golatkar et al. have recently shown that WD only matters at the start of the training in computer vision, upending traditional wisdom. Loshchilov et al. show that for adaptive optimizers, manually decaying weights can outperform adding an l2 penalty to the loss. This technique has become increasingly popular and is referred to as decoupled WD. The goal of this paper is to investigate these two recent empirical observations. We demonstrate that by applying WD only at the start, the network norm stays small throughout training. This has a regularizing effect as the effective gradient updates become larger. However, traditional generalizations metrics fail to capture this effect of WD, and we show how a simple scale-invariant metric can. We also show how the growth of network weights is heavily influenced by the dataset and its generalization properties. For decoupled WD, we perform experiments in NLP and RL where adaptive optimizers are the norm. We demonstrate that the primary issue that decoupled WD alleviates is the mixing of gradients from the objective function and the l2 penalty in the buffers of Adam (which stores the estimates of the first-order moment). Adaptivity itself is not problematic and decoupled WD ensures that the gradients from the l2 term cannot "drown out" the true objective, facilitating easier hyperparameter tuning.

General Cyclical Training of Neural Networks

“…This idea of allowing a hyper-parameter value to change during training (in replacement for a learning rate schedule) has been extended to other hyper-parameters, such as weight decay [13][14][15][16], and batch sizes [17]. Previous work [20], has demonstrated empirically a relationship between the optimal hyper-parameters of learning rate (LR), weight decay (WD), batch size (BS), and momentum (m) as…”

“…General cyclical training provides an intuitive understanding for the value of a one cycle training regime. Furthermore, this idea of allowing a hyper-parameter value to change during training has been extended to other hyperparameters, such as weight decay [13][14][15][16], and batch sizes [17].…”

Overview of convolutional neural networks architectures for brain tumor segmentation

“…Additionally, different layers are added to prevent the problem of overfitting, such as dropout layers and normalization layer that keeps the mean close to 0 and the standard deviation close to 1 for the output. This layer will hence accelerate training [13].…”

Structural hierarchical learning for energy networks

“…Batch normalization is a technique to standardize activations in intermediate layers of deep neural networks across minibatches. It has demonstrated improved accuracies and faster convergences due to its stabilization of the learning process [30]. Additionally, introducing batch normalization allows the in and outputs of the regression model to remain unscaled, thus retaining the hierarchical structure of the coherency-loss function.…”