SPeCiaL: Self-supervised Pretraining for Continual Learning

Caccia, Lucas; Pineau, Joëlle

doi:10.1007/978-3-031-17587-9_7

Cited by 21 publications

(41 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(i) Regularization-based approaches modify the classification objective to preserve previously learned representations or encourage more meaningful representations e.g. DER [4], ACE [5], and CoPE [13]. (ii) Sampling-based techniques focus on the optimal selection and storing of the most representative replay memory during online training, e.g.…”

Section: Related Workmentioning

confidence: 99%

“…Second, these benchmarks are incapable of measuring whether models can rapidly adapt to new data under a fast-changing distribution shift, which is one of the main problems in classical online learning literature [41]. There have been efforts to address the second limitation by using new metrics that measure test accuracy more frequently during training [5,24]. However, these metrics capture the adaptation to held-out test data rather than to the incoming future data.…”

Section: Related Workmentioning

confidence: 99%

“…Key Issues. As described earlier, the OCL evaluation in the literature [6,41] overlooks the training complexity in step (5). Under such a setup, all OCL methods have their parameters updated to θ t+1 before the stream S reveals images of the next step.…”

Section: Online Continual Learningmentioning

confidence: 99%

“…Figure 1 illustrates our proposed real-time evaluation. This is in contrast to all prior art [2,3,5] that (1) unreasonably allows an unlimited computational budget to train on any given stream data, and (2) unfairly compares OCL methods despite having different training complexity levels. Using our real-time evaluation protocol, we benchmark many existing OCL methods against a simple and inexpensive baseline, which mitigates forgetting by simply storing and replaying recently seen samples.…”

Section: Introductionmentioning

confidence: 99%

“…(1) We show that under our practical real-time evaluation, our simple baseline outperforms all the considered methods from the OCL literature, including recent SOTA approaches like ACE [5]. (2) We consider a complementary setup where the stream is as slow as the most trainingexpensive OCL method and compare that method against the compute-equivalent baseline.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Real-Time Evaluation in Online Continual Learning: A New Hope

Yasir¹,

Bibi²,

Alhamoud³

et al. 2023

Preprint

View full text Add to dashboard Cite

Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real-world setting, which motivates us to propose: a practical real-time evaluation of continual learning, in which the stream does not wait for the model to complete training before revealing the next data for predictions. To do this, we evaluate current CL methods with respect to their computational costs. We hypothesize that under this new evaluation paradigm, computationally demanding CL approaches may perform poorly on streams with a varying distribution. We conduct extensive experiments on CLOC, a large-scale dataset containing 39 million time-stamped images with geolocation labels. We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings. In addition, we explore various CL components commonly used in the literature, including memory sampling strategies and regularization approaches. We find that all considered methods fail to be competitive against our simple baseline. This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical. We hope that the evaluation we provide will be the first step towards a paradigm shift to consider the computational cost in the development of online continual learning methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Online Continual Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Real-Time Evaluation in Online Continual Learning: A New Hope

Yasir¹,

Bibi²,

Alhamoud³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Exemplar-Free Continual Representation Learning via Learnable Drift Compensation

Gomez-Villa,

Goswami,

Wang

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Aquaculture systems in Hanoi, Vietnam.

Nguyen¹

2020

Preprint

View full text Add to dashboard Cite

Hanoi, the capital of Vietnam, is one of the most densely populated areas in the world, with about 3,000 people km-2. Traditionally, aquaculture is practiced in ponds, reservoirs and urban lakes, rice fields, wastewater and VAC [an acronym from the Vietnamese words for garden (vuon), pond (ao) and livestock quarters (chuong)] systems. With over 5,000 ha of surface area and 3,000 ha of lowland rice fields, Hanoi has potential for further aquaculture development and tourism-entertainment services. The importance of aquaculture has long been recognized as a source of income and employment. Nowadays, pond aquaculture in peri-urban areas is decreasing due to urbanization, while in areas far from the city it is increasing through the conversion of lowland rice fields to fish ponds.

show abstract

SPeCiaL: Self-supervised Pretraining for Continual Learning

Cited by 21 publications

References 4 publications

Real-Time Evaluation in Online Continual Learning: A New Hope

Real-Time Evaluation in Online Continual Learning: A New Hope

Exemplar-Free Continual Representation Learning via Learnable Drift Compensation

Aquaculture systems in Hanoi, Vietnam.

Contact Info

Product

Resources

About