Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

Abstract

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a predefined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.Recent research has focused on solving these problems through various approaches, such as meta-learning [57, 34], regularization techniques [30], or knowledge distillation [38,6,62]. These methods have shown promising results in achieving incremental learning over time with a limited amount of data available. The general approaches consist in enhancing the basic representational capability of a predefined backbone architecture by adding special modules to entail backward compatibility with older classes during learning sessions. These solutions are computationally expensive since they need a large number of iterations in each session to adapt the additional modules to new classes while maintaining backward compatibility. Despite the high computational cost, they still fail to efficiently reduce the gap between the performance obtained on larger training sets

VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning

“…This problem is more serious in the few-shot scenario because of scarce training data. So, to alleviate the low-resource problem and improve the generalization ability of the model, meta learning [33][34][35][36][37][38] and few-shot learning [39][40][41][42][43][44][45][46][47][48][49][50][51][52] can be considered. Meta learning can construct a task pool to improve the generalization ability of the model.…”