Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Abstract

The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keep the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the re-initialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, or masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

Evolving and merging hebbian learning rules

“…The number of sub-networks -or potential winning tickets -dwindle rapidly if we decrease the size of the full network. To the best of our knowledge, the methods used for finding winning tickets [17,67] have not yet been explored in the case where the optimization method is ES, and much less in the context of indirect encoding. Our results hint that the Lottery Ticket Hypothesis might also hold in the indirect encoding setting that we employ here.…”

Putting 3D Spatially Sparse Networks on a Diet

“…Frankle et al [8] use larger architectures by relaxing the restriction of reverting the weights to initial values. Zhou et al [48] show that the difference between the initial weights and fine-tuned weights can be another pruning criterion. The proxies for determining lottery tickets in a data-efficient way have been recently studied extensively as well.…”

“…Once we finish training a spatially sparse convnet with dense weights, we prune some of the weights while preserving the test accuracy. In this paper, we utilize several pruning methods that belong to the family of magnitudebased pruning [7,11,20,48]. Specifically, we use three pruning methods with different pruning criteria and apply the criteria on the entire weights (global) or per layer (local).…”

“…We use three different network architectures with varying depth and width: Res16UNet14A, Res16UNet18A, Res16UNet34C [4]. We apply unstructured pruning [7,11,20,48] and compare the results with representative pointbased approaches [31,32,37,38,[43][44][45]47].…”

Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

“…Lottery Tickets. Lottery Tickets [7,8,41] is an interesting phenomenon: if we reset "salient weights" (trained weights with large magnitude) back to the values before optimization but after initialization, prune other weights (often > 90% of total weights) and retrain the model, the test performance is the same or better; if we reinitialize salient weights, the test performance is much worse. In our theory, the salient weights are those lucky regions (E j3 and E j4 in Fig.…”