The Lottery Ticket Hypothesis for Object Recognition

Girish, Sharath; Maiya, Shishira R; Gupta, Kamal Kumar; Chen, Hao; Davis, Larry S.; Shrivastava, Abhinav

doi:10.1109/cvpr46437.2021.00082

Cited by 40 publications

(37 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It achieves 32.7%AP on the COCO dataset when only keeping about 20% parameters of the original model. The superior performance over state-of-the-art result [7] confirms the effectiveness of our method.…”

Section: Introductionsupporting

confidence: 66%

“…In [1], the authors reveal that tickets found by the target object detection task surpass tickets found by image classification with a non-negligible margin. In [7], the authors check tickets found by supervised learning on an object detection dataset [20]. The result confirms that ImageNet tickets only transfer to a limited extent to downstream tasks, such as object detection or instance segmentation.…”

Section: Introductionmentioning

confidence: 72%

“…However, most of existing pruning algorithms only focus on the classification task defined on the source domain. Different from the strong transferability of the original model, a pruned network is hard to transfer to complicated downstream tasks such as object detection [7]. In this paper, we show that the image-level pretrain task is not capable of pruning models for diverse downstream tasks.…”

mentioning

confidence: 83%

See 2 more Smart Citations

Reconstruction Task Finds Universal Winning Tickets

Li¹,

Li²,

Qi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pruning well-trained neural networks is effective to achieve a promising accuracyefficiency trade-off in computer vision regimes. However, most of existing pruning algorithms only focus on the classification task defined on the source domain. Different from the strong transferability of the original model, a pruned network is hard to transfer to complicated downstream tasks such as object detection [7]. In this paper, we show that the image-level pretrain task is not capable of pruning models for diverse downstream tasks. To mitigate this problem, we introduce image reconstruction, a pixel-level task, into the traditional pruning framework. Concretely, an autoencoder is trained based on the original model, and then the pruning process is optimized with both autoencoder and classification losses. The empirical study on benchmark downstream tasks shows that the proposed method can outperform state-of-the-art results explicitly.Preprint. Under review.

show abstract

Section: Introductionsupporting

confidence: 66%

Section: Introductionmentioning

confidence: 72%

mentioning

confidence: 83%

See 1 more Smart Citation

Reconstruction Task Finds Universal Winning Tickets

Li¹,

Li²,

Qi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Lottery ticket hypothesis Following the pioneering work of Frankle & Carbin (2018), the search for lottery tickets has grown across several applications, such as language tasks, graph neural networks and federated learning (Chen et al, 2021;Li et al, 2020;Girish et al, 2021;Chen et al, 2020). While the LTH itself has yet to be proven mathematically, a somewhat strong version of it has been derived which shows that any target network can be approximated by pruning a randomly initialized network with minimal overparameterization (Malach et al, 2020;Pensia et al, 2020;Orseau et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Rare Gems: Finding Lottery Tickets at Initialization

Sreenivasan¹,

Sohn²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

It has been widely observed that large neural networks can be pruned to a small fraction of their original size, with little loss in accuracy, by typically following a time-consuming "train, prune, re-train" approach. Frankle & Carbin (2018) conjecture that we can avoid this by training lottery tickets, i.e., special sparse subnetworks found at initialization, that can be trained to high accuracy. However, a subsequent line of work presents concrete evidence that current algorithms for finding trainable networks at initialization, fail simple baseline comparisons, e.g., against training random sparse subnetworks. Finding lottery tickets that train to better accuracy compared to simple baselines remains an open problem. In this work, we partially resolve this open problem by discovering rare gems: subnetworks at initialization that attain considerable accuracy, even before training. Refining these rare gems-by means of finetuning-beats current baselines and leads to accuracy competitive or better than magnitude pruning methods.

show abstract

“…[6,21] further extend the lottery ticket hypothesis to a pre-trained BERT model. On object detection task, [22] proposes a guidance to find task-specific winning tickets for object detection, instance segmentation, and keypoint estimation. [23,24] have studied the lottery ticket hypothesis in unsupervised learning to reveal how well the tickets are transformed between different datasets.…”

Section: Related Workmentioning

confidence: 99%

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

Ma¹,

Yuan²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the "winning ticket" in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. † Equal contribution.Preprint. Under review.

show abstract

The Lottery Ticket Hypothesis for Object Recognition

Cited by 40 publications

References 15 publications

Reconstruction Task Finds Universal Winning Tickets

Reconstruction Task Finds Universal Winning Tickets

Rare Gems: Finding Lottery Tickets at Initialization

Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?

Contact Info

Product

Resources

About