Competence-based Curriculum Learning for Neural Machine Translation

Abstract

Current state-of-the-art NMT systems use large neural networks that are not only slow to train, but also often require many heuristics and optimization tricks, such as specialized learning rate schedules and large batch sizes. This is undesirable as it requires extensive hyperparameter tuning. In this paper, we propose a curriculum learning framework for NMT that reduces training time, reduces the need for specialized heuristics or large batch sizes, and results in overall better performance. Our framework consists of a principled way of deciding which training samples are shown to the model at different times during training, based on the estimated difficulty of a sample and the current competence of the model. Filtering training samples in this manner prevents the model from getting stuck in bad local optima, making it converge faster and reach a better solution than the common approach of uniformly sampling training examples. Furthermore, the proposed method can be easily applied to existing NMT models by simply modifying their input data pipelines. We show that our framework can help improve the training time and the performance of both recurrent neural network models and Transformers, achieving up to a 70% decrease in training time, while at the same time obtaining accuracy improvements of up to 2.2 BLEU.

Curriculum-Based Transfer Learning for an Effective End-to-End Spoken Language Understanding and Domain Portability

“…Thanks to the use of a transfer learning approach inspired by curriculum learning [20,10], we are now able to reach stateof-the-art performance. A curriculum learning approach has also been recently proposed with success for machine translation in [21]. At the end, we also address the issue of domain portability for SLU systems [13,22] that can obviously be tackled as a transfer learning problem.…”

Curriculum Learning Strategies for IR

“…training the model using easy instances first and increasing the difficulty during the training procedure, has been empirically demonstrated in different machine learning problems, e.g. image classification [11,14], machine translation [21,30,60] and answer generation [23].…”

“…Sentence length Machine translation [30], language generation [42], reading comprehension [58] Word rarity Machine translation [30,60], language modeling [1] External model confidence Machine translation [60], image classification [14,49], ad-hoc retrieval [9] Supervision signal intensity Facial expression recognition [12], ad-hoc retrieval [9] Noise estimate Speaker identification [34], image classification [5] Human annotation Image classification [45] (through weak supervision)…”

“…where k N , and perform an optimization procedure sequentially in {B 1 , ..., B M }. The CL framework employed here is inspired by previous works [30,49]. It is defined by two functions: the scoring function which determines the difficulty of instances and the pacing function which controls the pace with which to transition from easy to hard instances during training.…”

“…Assuming that we know the difficulty of each instance in our training set, we still need to define how are we going to transition from easy to hard instances. We use the concept of pacing functions f pace (s); they should each have the following properties [30,49]: (i) start at an initial value of training instances f pace (0) = δ with δ > 0, so that the model has a number of instances to train in the first iteration, (ii) be non-decreasing, so that harder instances are added to the training set, and, (iii) eventually all instances are available for sampling when it reaches T iterations, f pace (T ) = 1.…”