Task Adaptive Parameter Sharing for Multi-Task Learning

Wallingford, Matthew; Li, Hao; Achille, Alessandro; Ravichandran, Avinash; Fowlkes, Charless C.; Bhotika, Rahul; Soatto, Stefano

doi:10.48550/arxiv.2203.16708

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works in efficient neural network literature [9,84,91] aim at packing neural networks of varying capacity within the same larger network. However, the weights for each progressively smaller network can be different and often require distinct forward passes to isolate the final representations.…”

Section: Related Workmentioning

confidence: 99%

Matryoshka Representations for Adaptive Deployment

Kusupati¹,

Bhatt²,

Wallingford³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Learned representations are a central component in modern ML systems, serving a multitude of downstream tasks. When training such representations, it is often the case that computational and statistical constraints for each downstream task are unknown. In this context, rigid fixed-capacity representations can be either over or under-accommodating to the task at hand. This leads us to ask: can we design a flexible representation that can adapt to multiple downstream tasks with varying computational resources? Our main contribution is Matryoshka Representation Learning (MRL) which encodes information at different granularities and allows a single embedding to adapt to the computational constraints of downstream tasks. MRL minimally modifies existing representation learning pipelines and imposes no additional cost during inference and deployment. MRL learns coarse-to-fine representations that are at least as accurate and rich as independently trained low-dimensional representations. The flexibility within the learned Matryoshka Representations offer: (a) up to 14× smaller embedding size for ImageNet-1K classification at the same level of accuracy; (b) up to 14× real-world speed-ups for large-scale retrieval on ImageNet-1K and 4K; and (c) up to 2% accuracy improvements for long-tail few-shot classification, all while being as robust as the original representations. Finally, we show that MRL extends seamlessly to web-scale datasets (ImageNet, JFT) across various modalities -vision (ViT, ResNet), vision + language (ALIGN) and language (BERT). MRL code and pretrained models are open-sourced at https://github.com/RAIVNLab/MRL. * Equal contribution -AK led the project with extensive support from GB and AR for experimentation.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%