Linear Mode Connectivity and the Lottery Ticket Hypothesis

Frankle, Jonathan; Dziugaite, Gintare Karolina; Roy, Daniel M.; Carbin, Michael

doi:10.48550/arxiv.1912.05671

Cited by 15 publications

(24 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In other words, we could have trained smaller networks from the start if only we had known which subnetworks to choose. Unfortunately, LTH requires to empirically find these intriguing subnetworks by an iterative pruning [17,18] , which still cannot get rid of the expensiveness of post-training pruning. In view of that, follow-up works reveal that sparsity patterns might emerge at the initialization [19,20], the early stage of training [21,22], or in dynamic forms throughout training [23][24][25] by updating model parameters and architecture typologies simultaneously.…”

Section: Introductionmentioning

confidence: 99%

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chen¹,

Gan²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Vision transformers (ViTs) have recently received explosive popularity, but their enormous model sizes and training costs remain daunting. Conventional posttraining pruning often incurs higher training budgets. In contrast, this paper aims to trim down both the training memory overhead and the inference complexity, without sacrificing the achievable accuracy. We launch and report the first-ofits-kind comprehensive exploration, on taking a unified approach of integrating sparsity in ViTs "from end to end". Specifically, instead of training full ViTs, we dynamically extract and train sparse subnetworks, while sticking to a fixed small parameter budget. Our approach jointly optimizes model parameters and explores connectivity throughout training, ending up with one sparse network as the final output. The approach is seamlessly extended from unstructured to structured sparsity, the latter by considering to guide the prune-and-grow of self-attention heads inside ViTs. For additional efficiency gains, we further co-explore data and architecture sparsity, by plugging in a novel learnable token selector to adaptively determine the currently most vital patches. Extensive results on ImageNet with diverse ViT backbones validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization. Perhaps most surprisingly, we find that the proposed sparse (co-)training can even improve the ViT accuracy rather than compromising it, making sparsity a tantalizing "free lunch". For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs and 4.40% running time savings. Our codes are available at https://github.com/VITA-Group/SViTE.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 99%

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chen¹,

Gan²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, Draxler et al (2018); Garipov et al (2018) show that local minima found by stochastic gradient descent (SGD) can be connected via piecewise linear paths. Frankle et al (2019) further show that linearly connected solutions may be found if networks share the same initialization. demonstrate the connection between linear connectivity and the advantage nonlinear networks enjoy over their linearized version.…”

Section: Related Workmentioning

confidence: 83%

Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

Lucas,

Bae,

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective. This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014), persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks. Extending this work, we evaluate several hypotheses for this property that, to our knowledge, have not yet been explored. Using tools from differential geometry, we draw connections between the interpolated paths in function space and the monotonicity of the network -providing sufficient conditions for the MLI property under mean squared error. While the MLI property holds under various settings (e.g. network architectures and learning problems), we show in practice that networks violating the MLI property can be produced systematically, by encouraging the weights to move far from initialization. The MLI property raises important questions about the loss landscape geometry of neural networks and highlights the need to further study their global properties.

show abstract

“…5 Magnitude pruning avoids layer-collapse with conservation and iteration Having demonstrated and investigated the cause of layer-collapse in single-shot pruning methods at initialization, we now explore an iterative pruning method that appears to avoid the issue entirely. Iterative Magnitude Pruning (IMP) is a recently proposed pruning algorithm that has proven to be successful in finding extremely sparse trainable neural networks at initialization (winning lottery tickets) [10,11,12,41,42,43,44]. The algorithm follows three simple steps.…”

Section: Synflow Random Graspmentioning

confidence: 99%

Pruning neural networks without any data by iteratively conserving synaptic flow

Tanaka¹,

Kunin²,

Yamins³

et al. 2020

Preprint

View full text Add to dashboard Cite

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradientbased pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.9 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that data must be used to quantify which synapses are important.

show abstract

Linear Mode Connectivity and the Lottery Ticket Hypothesis

Cited by 15 publications

References 0 publications

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

Pruning neural networks without any data by iteratively conserving synaptic flow

Contact Info

Product

Resources

About