Deep Bayesian Unsupervised Lifelong Learning

Zhao, Tingting; Wang, Zifeng; Masoomi, Aria; Dy, Jennifer G.

doi:10.1016/j.neunet.2022.02.001

Cited by 21 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Architecture-based methods assign isolated parameters for each task. These methods can be further categorized as expanding the model [51,65,26,31,59], or dividing the model [35,52,58,17,11]. However, a major part of the work is limited to the task-incremental setting [52,35,34,17], while other work only considers specific convolutional-based architectures [61,44,11].…”

Section: Related Workmentioning

confidence: 99%

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

Wang¹,

Zhang²,

Ebrahimi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Continual learning aims to enable a single model to learn a sequence of tasks without catastrophic forgetting. Top-performing methods usually require a rehearsal buffer to store past pristine examples for experience replay, which, however, limits their practical value due to privacy and memory constraints. In this work, we present a simple yet effective framework, DualPrompt, which learns a tiny set of parameters, called prompts, to properly instruct a pre-trained model to learn tasks arriving sequentially without buffering past examples. DualPrompt presents a novel approach to attach complementary prompts to the pre-trained backbone, and then formulates the objective as learning task-invariant and task-specific "instructions". With extensive experimental validation, DualPrompt consistently sets state-of-the-art performance under the challenging class-incremental setting. In particular, DualPrompt outperforms recent advanced continual learning methods with relatively large buffer sizes. We also introduce a more challenging benchmark, Split ImageNet-R, to help generalize rehearsal-free continual learning research. Source code is available at https://github.com/google-research/l2p.

show abstract

Section: Related Workmentioning

confidence: 99%

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

Wang¹,

Zhang²,

Ebrahimi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The input of the DPMM module is the latent variable C of the biological information, where c n is the n-th component in C. The prior distribution provided by the DPMM module can constrain c n and infer the clustering label y n of cell n. In this module, c n is assumed to be generated from the Dirichlet process mixture model, and the loss function of this module is defined as the negative log-likelihood of c n and the distribution determined by DPMM [14]. The loss function l DPMM is defined as the following:…”

Section: The Dpmm Modulementioning

confidence: 99%

A Joint Batch Correction and Adaptive Clustering Method of Single-Cell Transcriptomic Data

An,

Shi,

Liu

et al. 2023

Mathematics

View full text Add to dashboard Cite

Clustering analysis for single-cell RNA sequencing (scRNA-seq) data is essential for characterizing cellular heterogeneity. However, batch information caused by batch effects is often confused with the intrinsic biological information in scRNA-seq data, which makes accurate clustering quite challenging. A Deep Adaptive Clustering with Adversarial Learning method (DACAL) is proposed here. DACAL jointly optimizes the batch correcting and clustering processes to remove batch effects while retaining biological information. DACAL achieves batch correction and adaptive clustering without requiring manually specified cell types or resolution parameters. DACAL is compared with other widely used batch correction and clustering methods on human pancreas datasets from different sequencing platforms and mouse mammary datasets from different laboratories. The results demonstrate that DACAL can correct batch effects efficiently and adaptively find accurate cell types, outperforming competing methods. Moreover, it can obtain cell subtypes with biological meanings.

show abstract

“…However, their performance degrades for smaller memory size (Cha et al, 2021), and storing these exemplars can introduce security and privacy concerns Shokri & Shmatikov (2015). Architecture-driven CL methods either dynamically expand a network Rusu et al (2016); Li et al (2019b); Zhao et al (2022), or divide into sub-networks to cater for the new tasks (Zhao et al, 2022;Wang et al, 2020;Ke et al, 2020;Rajasegaran et al, 2019a). Such approaches lack scalability, since the network capacity grows with tasks.…”

Section: Related Workmentioning

confidence: 99%

CLIP model is an Efficient Continual Learner

Thengane¹,

Khan²,

Hayat³

et al. 2022

Preprint

View full text Add to dashboard Cite

The continual learning setting aims to learn new tasks over time without forgetting the previous ones. The literature reports several significant efforts to tackle this problem with limited or no access to previous task data. Among such efforts, typical solutions offer sophisticated techniques involving memory replay, knowledge distillation, model regularization, and dynamic network expansion. The resulting methods have a retraining cost at each learning task, dedicated memory requirements, and setting-specific design choices. In this work, we show that a frozen CLIP (Contrastive Language-Image Pretraining) model offers astounding continual learning performance without any fine-tuning (zero-shot evaluation). We evaluate CLIP under a variety of settings including class-incremental, domain-incremental and task-agnostic incremental learning on five popular benchmarks (ImageNet-100 & 1K, CORe50, CIFAR-100, and TinyImageNet). Without any bells and whistles, the CLIP model outperforms the state-of-the-art continual learning approaches in majority of the settings. We show the effect on CLIP model's performance by varying text inputs with simple prompt templates. To the best of our knowledge, this is the first work to report the CLIP zero-shot performance in a continual setting. We advocate the use of this strong yet embarrassingly simple baseline for future comparisons in the continual learning tasks. Code is available at https://github.com/vgthengane/Continual-CLIP.

show abstract

Deep Bayesian Unsupervised Lifelong Learning

Cited by 21 publications

References 22 publications

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

A Joint Batch Correction and Adaptive Clustering Method of Single-Cell Transcriptomic Data

CLIP model is an Efficient Continual Learner

Contact Info

Product

Resources

About