Energy-based Latent Aligner for Incremental Learning

Joseph, K J; Khan, Salman; Khan, Fahad Shahbaz; Anwer, Rao Muhammad; Balasubramanian, Vineeth N

doi:10.1109/cvpr52688.2022.00730

Cited by 17 publications

(6 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regularized-based methods [20]- [24] strive to search the important parameters to the original tasks and constrain their changes in the coming tasks. Elastic Weight Consolidation (EWC) [20] computes the importance of each parameter via a diagonal approximation of the Fisher Information Matrix, and slows down the learning of each parameter selectively.…”

Section: Parametric Methodsmentioning

confidence: 99%

“…Riemania Walk (RWalk) [23] calculates the parameter importance by fusing Fisher Information Matrix approximation and online path integral with a theoretically grounded KL-divergence based perspective. ELI [24] learns an energy manifold for the latent representations to counter the representational shift during Incremental learning. The advantage of the regularized-based method is that they do not need to store the samples of old tasks, i.e., exemplar samples.…”

Section: Parametric Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Margin-Based Adversarial Joint Alignment Domain Adaptation

Zuo

Yao

Zhuang

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Audio-visual video recognition (AVVR) aims to integrate audio and visual clues to categorize videos accurately. While existing methods train AVVR models using provided datasets and achieve satisfactory results, they struggle to retain historical class knowledge when confronted with new classes in real-world situations. Currently, there are no dedicated methods for addressing this problem, so this paper concentrates on exploring Class Incremental Audio-Visual Video Recognition (CIAVVR). For CIAVVR, since both stored data and learned model of past classes contain historical knowledge, the core challenge is how to capture past data knowledge and past model knowledge to prevent catastrophic forgetting. As audio-visual data and model inherently contain hierarchical structures, i.e., model embodies low-level and high-level semantic information, and data comprises snippet-level, video-level, and distribution-level spatial information, it is essential to fully exploit the hierarchical data structure for data knowledge preservation and hierarchical model structure for model knowledge preservation. However, current image class incremental learning methods do not explicitly consider these hierarchical structures in model and data. Consequently, we introduce Hierarchical Augmentation and Distillation (HAD), which comprises the Hierarchical Augmentation Module (HAM) and Hierarchical Distillation Module (HDM) to efficiently utilize the hierarchical structure of data and models, respectively. Specifically, HAM implements a novel augmentation strategy, segmental feature augmentation, to preserve hierarchical model knowledge. Meanwhile, HDM introduces newly designed hierarchical (video-distribution) logical distillation and hierarchical (snippet-video) correlative distillation to capture and maintain the hierarchical intra-sample knowledge of each data and the hierarchical inter-sample knowledge between data, respectively. Evaluations on four benchmarks (AVE, AVK-100, AVK-200, and AVK-400) demonstrate that the proposed HAD effectively captures hierarchical information in both data and models, resulting in better preservation of historical class knowledge and improved performance. Furthermore, we provide a theoretical analysis to support the necessity of the segmental feature augmentation strategy.

show abstract

Section: Parametric Methodsmentioning

confidence: 99%

Section: Parametric Methodsmentioning

confidence: 99%

Margin-Based Adversarial Joint Alignment Domain Adaptation

Zuo

Yao

Zhuang

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…There are three main lines of work to address the stability-plasticity trade-off in CIL. Distillation-based methods [8,15,16,21,27,37,41] introduce different knowledge distillation (KD) losses to consolidate previous knowledge when training the model on new data. The key idea is to enforce model prediction logits [21,30], feature maps [8,14], or topologies in the feature space [36] to be close to those of the pre-phase model.…”

Section: Related Workmentioning

confidence: 99%

Online Hyperparameter Optimization for Class-Incremental Learning

Liu¹,

Li²,

Schiele³

et al. 2023

Preprint

View full text Add to dashboard Cite

Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phaseby-phase. An inherent challenge of CIL is the stabilityplasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. However, none of the existing CIL models can achieve the optimal tradeoff in different data-receiving settingswhere typically the training-from-half (TFH) setting needs more stability, but the training-from-scratch (TFS) needs more plasticity. To this end, we design an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori. Specifically, we first introduce the key hyperparameters that influence the tradeoff, e.g., knowledge distillation (KD) loss weights, learning rates, and classifier types. Then, we formulate the hyperparameter optimization process as an online Markov Decision Process (MDP) problem and propose a specific algorithm to solve it. We apply local estimated rewards and a classic bandit algorithm Exp3 [4] to address the issues when applying online MDP methods to the CIL protocol. Our method consistently improves top-performing CIL methods in both TFH and TFS settings, e.g., boosting the average accuracy of TFH and TFS by 2.2 percentage points on ImageNet-Full, compared to the state-of-the-art [23]. 1

show abstract

“…The pioneering work is LwF [6], which proposes an incremental learning method without catastrophic forgetting and conducts extensive experiments. In follow-up studies, researchers try various incremental learning strategies including replay-based methods [35], [36], [37], architecturebased methods [38], [39], [40], and regularization-based methods [41], [42], [43]. Replay-based incremental learning means retaining the exemplars of the old classes in the new task learning.…”

Section: B Incremental Learningmentioning

confidence: 99%

Class-Prompting Transformer for Incremental Semantic Segmentation

Song,

Shi,

Shang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Class-incremental Semantic Segmentation (CISS) aims to learn new tasks sequentially that assign a specific category to each pixel of a given image while preserving the original capability to segment the old classes even if the labels of old tasks are absent. Most existing CISS methods suppress catastrophic forgetting by directly distilling on specific layers, which ignores the semantic gap between training data from the old and new classes with different distributions and leads to distillation errors, thus affecting segmentation performance. In this paper, we propose a Class-prompting Transformer (CPT) to introduce external prior knowledge provided by a pre-trained vision-language encoder into CISS pipelines for bridging the old and new classes and performing more generalized initialization and distillation. Specifically, we proposed a Prompt-guided Initialization Module (PIM), which measures the relationships between the class prompts and old query parameters to initialize the new query parameters for relocating the previous knowledge to the learning of new tasks. Then, a Semantic-aligned Distillation Module (SDM) is proposed to incorporate class prompt information with the class-aware embeddings extracted from the decoder to prevent the semantic gap problem between distinct class data and conduct adaptive knowledge transfer to suppress catastrophic forgetting. Extensive experiments on Pascal VOC and ADE20K datasets for the CISS task demonstrate the superiority of the proposed method, which achieves state-of-the-art performance.INDEX TERMS Incremental semantic segmentation, knowledge distillation, class prompt learning.LINFENG XU (Member, IEEE) received the Ph.D. degree in signal and information processing from the School of Electronic Engineering,

show abstract

Energy-based Latent Aligner for Incremental Learning

Cited by 17 publications

References 21 publications

Margin-Based Adversarial Joint Alignment Domain Adaptation

Margin-Based Adversarial Joint Alignment Domain Adaptation

Online Hyperparameter Optimization for Class-Incremental Learning

Class-Prompting Transformer for Incremental Semantic Segmentation

Contact Info

Product

Resources

About