ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition

FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification

“…Second, we evaluate FIT against BiT on the challenging VTAB-1k benchmark [31], where BiT has been shown to outperform all meta-learners [10,28]. Third, we show how FIT can be used in a personalization scenario on the ORBIT [5] dataset, where a smaller updateable model is an important evaluation metric. Finally, we apply FIT to a few-shot federated learning scenario where minimizing the number of parameter updates and their size is a key requirement.…”

“…In our experiments, we use ORBIT [5], a real-world few-shot video dataset recorded by people who are blind/low-vision. A blind or vision-impaired user collects a series of short videos on their smartphone of objects that they would like to recognize.…”

“…The personalization experiments were carried out on a single NVIDIA GeForce RTX 3090 with 24GB of memory. It takes approximately 5 hours to train FIT-LDA personalization models for all the ORBIT [5] test tasks. We derived all hyperparameters empirically from a small number of runs.…”

“…With the success of the commercial application of deep learning in many fields such as computer vision [1], natural language processing [2], speech recognition [3], and language translation [4], an increasing number of models are being trained on central servers and then deployed on remote devices, often to personalize a model to a specific user's needs. Personalization requires models that can be updated inexpensively by minimizing the number of parameters that need to be stored and/or transmitted and frequently calls for few-shot learning methods as the amount of training data from an individual user may be small [5]. At the same time, for privacy, security, and performance reasons, it can be advantageous to use federated learning where a model is trained on an array of remote devices, each with different data, and share gradient or parameter updates instead of training data with a central server [6].…”

“…In this work, we focus on designing deep learning network architectures and associated training protocols that allow image classification models to be updated with only a small subset of the total model parameters, without sacrificing prediction performance when there is only a small number of training examples available. This leads to reduced storage and transmission costs for updating personalized models on remote devices [5], distributed training in federated learning [6], and efficient ensemble realization [13], among other applications. To realize our goal of small model updates, we pursue a transfer learning approach that takes advantage of image classification backbones that have been pretrained on large upstream datasets [12].…”