GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Abstract

Standardized benchmarks drive progress in machine learning. However, with repeated testing, the risk of overfitting grows as algorithms overexploit benchmark idiosyncrasies. In our work, we seek to mitigate this challenge by compiling ever-expanding large-scale benchmarks called Lifelong Benchmarks. As exemplars of our approach, we create Lifelong-CIFAR10 and Lifelong-ImageNet, containing (for now) 1.69M and 1.98M test samples, respectively. While reducing overfitting, lifelong benchmarks introduce a key challenge: the high cost of evaluating a growing number of models across an ever-expanding sample set. To address this challenge, we also introduce an efficient evaluation framework: Sort & Search (S&S), which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples, enabling cost-effective lifelong benchmarking. Extensive empirical evaluations across ∼31,000 models demonstrate that S&S achieves highly-efficient approximate accuracy measurement, reducing compute cost from 180 GPU days to 5 GPU hours (∼1000x reduction) on a single A100 GPU, with low approximation error. As such, lifelong benchmarks offer a robust, practical solution to the "benchmark exhaustion" problem.

Towards Realistic Single-Task Continuous Learning Research for NER

“…The latter is considered a strong CL baseline (Maltoni and Lomonaco, 2019) and thus we use this approach in this study. We also compare against a variation of the replay-based GDumb baseline (Prabhu et al, 2020). GDumb collects examples into a memory buffer with a limited budget size k, balancing the distribution over labels by greedily sampling underrepresented label types and ejecting over-sampled label types.…”

CILEA-NET: Curriculum-Based Incremental Learning Framework for Remote Sensing Image Classification

“…The approach in [28] tries to retrieve only the samples that are most conflicted. At the same time, [29] proposes to improve performance by greedily storing samples in memory and retraining on these stored samples while testing. The work in [30] proposes an expansion-based approach for task-free continual learning built upon the Bayesian nonparametric.…”

“…Comparison is also performed with [27], which proposes an intermediate expert to adapt the target model to the new task and [28], which retrieves the samples that are frequently conflicted. We also include our comparison with [29], which greedily stores samples in memory and trains a model from scratch and uses these samples during testing. Finally, we compare with [30], which increases the number of neural network experts under the Bayesian non-parametric framework We carried out experiments for comparing the results obtained from the proposed approach with different algorithms: 1.…”

“…RS refers to the replay-based work proposed in [31]. Please note, that DR, MIR, GDUMB, CN-DPM are also used for comparison which refer to the works proposed in [27], [28], [29], and [30] respectively. We bestow on the extensive work presented in [44] and the code provided here 1 to replicate these results.…”

Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates

“…Our work follows this transfer learning paradigm but our main focus is to investigate the regression phenomenon when updating backbone pre-trained models. Another related stream of research is lifelong learning (Lopez-Paz and Ranzato, 2017;Yoon et al, 2018;Delange et al, 2021;Sun et al, 2019;Chuang et al, 2020), incremental learning (Rebuffi et al, 2017Chaudhry et al, 2018;Prabhu et al, 2020), or concept drifting (Schlimmer and Granger, 1986;Tsymbal, 2004;Klinkenberg, 2005;Žliobaitė I., 2016) which aims to accumulate knowledge learned either in previous tasks or from data with changing distribution. The model update regression problem differs in that models are trained on the same task and dataset, but we update from one model to another.…”