On the Transferability of Winning Tickets in Non-Natural Image Datasets

Abstract

We study the generalization properties of pruned neural networks that are the winners of the lottery ticket hypothesis on datasets of natural images. We analyse their potential under conditions in which training data is scarce and comes from a non-natural domain. Specifically, we investigate whether pruned models that are found on the popular CIFAR-10/100 and Fashion-MNIST datasets, generalize to seven different datasets that come from the fields of digital pathology and digital heritage. Our results show that there are significant benefits in transferring and training sparse architectures over larger parametrized models, since in all of our experiments pruned networks, winners of the lottery ticket hypothesis, significantly outperform their larger unpruned counterparts. These results suggest that winning initializations do contain inductive biases that are generic to some extent, although, as reported by our experiments on the biomedical datasets, their generalization properties can be more limiting than what has been so far observed in the literature.

Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding

“…We call lottery tickets sourced on a specific task bespoke lottery tickets. We confirm previous findings of ticket transferability [39,45], and we aim to analyze whether similarity of masks can trivially explain this property.…”

“…On the other hand, lottery tickets have been shown to possess generalization properties that allow for their reuse across similar tasks, thus reducing the computational cost of finding dataset-dependent sparse sub-networks [38,45]. However, the mechanism that powers their transferability properties is not yet clearly understood, and is therefore subject to further study in this work.…”

“…Additionally, preliminary evidence for LT transferability has been provided in both natural and non-natural image domains [38,45]. In the context of LTs, transfer refers to the ability to retrain a sub-network on a new task.…”

Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win

“…Frankle et al ( , 2020a proposed late rewinding as a solution. Morcos et al (2019); Sabatelli, Kestemont, and Geurts (2020) showed that LTs trained on large datasets transfer to smaller ones, but not vice versa. Mostafa and Wang (2019) propose replacing low magnitude parameters with random connections and report improved generalization.…”

Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient

“…Soelen et al [24] show that the winning tickets from one task are transferable to another related task. Sabatelli et al [25] show further that the trained winning tickets can be transferred with minimal retraining on new tasks, and sometimes may even generalize better than models trained from scratch specifically for the new task.…”