When Do Curricula Work?

Wu, Xiaoxia; Dyer, Ethan; Neyshabur, Behnam

doi:10.48550/arxiv.2012.03107

Cited by 9 publications

(13 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared popular CL methods on a large number of datasets and deep learning architectures in the text and image domains, and we found that current CL methods do not increase success much, in line with the results of studies [4,11] recently published in the literature. Almost all of the methods in the literature increase the size of the dataset [3,4,6].…”

Section: Introductionsupporting

confidence: 81%

“…In [11], Extensive experiments were conducted on curriculum learning and examined the situations in which curriculum methods work. Curriculum, Anti-Curriculum, and Random-Curriculum methods were compared.…”

Section: Literature Reviewmentioning

confidence: 99%

“…There are studies that have achieved successful results using curriculum [3,4,6,7], anti-curriculum [8][9][10] and randomcurriculum [11] . The common point of these studies is to achieve more successful results in training by increasing the size of the training set.…”

Section: Introductionmentioning

confidence: 99%

“…Randomly ordered samples can perform as well as or better than curricula and anti-curricula, and better results can be obtained as the training dataset dynamically increases and decreases over time, implying that any benefit is entirely due to the dynamic training set size [11] .…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Cyclical Curriculum Learning

Toprak¹,

Amasyalı²

2022

Preprint

View full text Add to dashboard Cite

Artificial neural networks (ANN) are inspired by human learning. However, unlike human education, classical ANN does not use a curriculum. Curriculum Learning (CL) refers to the process of ANN training in which examples are used in a meaningful order. When using CL, training begins with a subset of the dataset and new samples are added throughout the training, or training begins with the entire dataset and the number of samples used is reduced. With these changes in training dataset size, better results can be obtained with curriculum, anti-curriculum, or random-curriculum methods than the vanilla method. However, a generally efficient CL method for various architectures and data sets is not found. In this paper, we propose cyclical curriculum learning (CCL), in which the data size used during training changes cyclically rather than simply increasing or decreasing. Instead of using only the vanilla method or only the curriculum method, using both methods cyclically like in CCL provides more successful results. We tested the method on 18 different data sets and 15 architectures in image and text classification tasks and obtained more successful results than no-CL and existing CL methods. We also have shown theoretically that it is less erroneous to apply CL and vanilla cyclically instead of using only CL or only vanilla method. The code of Cyclical Curriculum is available at https://github.com/CyclicalCurriculum/Cyclical-Curriculum.

show abstract

Section: Introductionsupporting

confidence: 81%

Section: Literature Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cyclical Curriculum Learning

Toprak¹,

Amasyalı²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our goal is to define metrics to detect underrepresented data regions during the training. In supervised learning, training dynamics-the behavior of a model as training progresses-have been used to detect "hard-to-learn" samples (Chang et al, 2017;Swayamdipta et al, 2020;Wu et al, 2020a).…”

Section: New Metrics To Diagnose Gan Trainingmentioning

confidence: 99%

Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks

Lee¹,

Kim²,

Hong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite remarkable performance in producing realistic samples, Generative Adversarial Networks (GANs) often produce low-quality samples near low-density regions of the data manifold. Recently, many techniques have been developed to improve the quality of generated samples, either by rejecting low-quality samples after training or by pre-processing the empirical data distribution before training, but at the cost of reduced diversity. To guarantee both the quality and the diversity, we propose a simple yet effective method to diagnose and emphasize underrepresented samples during training of a GAN. The main idea is to use the statistics of the discrepancy between the data distribution and the model distribution at each data instance. Based on the observation that the underrepresented samples have a high average discrepancy or high variability in discrepancy, we propose a method to emphasize those samples during training of a GAN. Our experimental results demonstrate that the proposed method improves GAN performance on various datasets, and it is especially effective in improving the quality of generated samples with minor features.

show abstract

Improving image classification of gastrointestinal endoscopy using curriculum self-supervised learning

Guo,

Somayajula,

Hosseini

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Endoscopy, a widely used medical procedure for examining the gastrointestinal (GI) tract to detect potential disorders, poses challenges in manual diagnosis due to non-specific symptoms and difficulties in accessing affected areas. While supervised machine learning models have proven effective in assisting clinical diagnosis of GI disorders, the scarcity of image-label pairs created by medical experts limits their availability. To address these limitations, we propose a curriculum self-supervised learning framework inspired by human curriculum learning. Our approach leverages the HyperKvasir dataset, which comprises 100k unlabeled GI images for pre-training and 10k labeled GI images for fine-tuning. By adopting our proposed method, we achieved an impressive top-1 accuracy of 88.92% and an F1 score of 73.39%. This represents a 2.1% increase over vanilla SimSiam for the top-1 accuracy and a 1.9% increase for the F1 score. The combination of self-supervised learning and a curriculum-based approach demonstrates the efficacy of our framework in advancing the diagnosis of GI disorders. Our study highlights the potential of curriculum self-supervised learning in utilizing unlabeled GI tract images to improve the diagnosis of GI disorders, paving the way for more accurate and efficient diagnosis in GI endoscopy.

show abstract

When Do Curricula Work?

Cited by 9 publications

References 28 publications

Cyclical Curriculum Learning

Cyclical Curriculum Learning

Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks

Improving image classification of gastrointestinal endoscopy using curriculum self-supervised learning

Contact Info

Product

Resources

About