Data Augmentation Approaches in Natural Language Processing: A Survey

Li, Bohan; Hou, Yutai; Che, Wanxiang

doi:10.48550/arxiv.2110.01852

Cited by 5 publications

(7 citation statements)

References 78 publications

(186 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The noising-based data augmentation methods can not only improve the translation quality of the model, but also improve its robustness [Miyato et al, 2017;Li et al, 2021]. To test robustness on noisy inputs, similarly to Cheng et al…”

Section: Results To Noisy Inputsmentioning

confidence: 99%

“…Data augmentation are techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data [Li et al, 2021]. These methods can significantly boost the accuracy of deep learning methods.…”

Section: Introductionmentioning

confidence: 99%

“…According to the diversity of augmented data, Li et al [2021] frame data augmentation methods into three categories, including paraphrasing, noising, and sampling.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

Chang¹,

Qiu²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

In Neural Machine Translation (NMT), data augmentation methods such as back-translation have proven their effectiveness in improving translation performance. In this paper, we propose a novel data augmentation approach for NMT, which is independent of any additional training data. Our approach, AdMix, consists of two parts: 1) introduce faint discrete noise (word replacement, word dropping, word swapping) into the original sentence pairs to form augmented samples; 2) generate new synthetic training data by softly mixing the augmented samples with their original samples in training corpus. Experiments on three translation datasets of different scales show that AdMix achieves significant improvements (1.0 to 2.7 BLEU points) over strong Transformer baseline. When combined with other data augmentation techniques (e.g., back-translation), our approach can obtain further improvements.

show abstract

Section: Results To Noisy Inputsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

Chang¹,

Qiu²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the data format varies between different downstream tasks, there is still a gap for data processing methods between tasks. Therefore, currently, no universal method is effective for all NLG tasks [96]. Data pre-processing might result in grammatical errors or semantic transformation between the original and processed data, which can negatively affect the performance of generation.…”

Section: Future Directions In Mitigation Methodsmentioning

confidence: 99%

Survey of Hallucination in Natural Language Generation

Ji¹,

Lee²,

Frieske³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent natural language generation, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended texts, which degrades the system performance and fail to meet user expectations in many real-world scenarios. In order to address this issue, there have been studies in measuring and mitigating hallucinated texts. However there has not been a comprehensive review of the state-of-the-art in hallucination detection and mitigation.In this survey, we provide a broad overview of the research progress and challenges in the hallucination problem of NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; (2) an overview of task-specific research progress for hallucinations in a large set of downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, and machine translation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.

show abstract

“…Data augmentation (DA), a technique for increasing the number of data, is applied in many deep learning applications, for example, in image [18] including major convolutional neural network architectures such as [12] and successors, speech [17], and natural language processing [13]. Despite the various reports that demonstrate the effectiveness of DA, the working mechanism is not clear.…”

Section: Background and Motivationmentioning

confidence: 99%

On Gradient Descent Training Under Data Augmentation with On-Line Noisy Copies

HAGIWARA

2023

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

In machine learning, data augmentation (DA) is a technique for improving the generalization performance of models. In this paper, we mainly consider gradient descent of linear regression under DA using noisy copies of datasets, in which noise is injected into inputs. We analyze the situation where noisy copies are newly generated and injected into inputs at each epoch, i.e., the case of using on-line noisy copies. Therefore, this article can also be viewed as an analysis on a method using noise injection into a training process by DA. We considered the training process under three training situations which are the full-batch training under the sum of squared errors, and full-batch and mini-batch training under the mean squared error. We showed that, in all cases, training for DA with on-line copies is approximately equivalent to the 2 regularization training for which variance of injected noise is important, whereas the number of copies is not. Moreover, we showed that DA with on-line copies apparently leads to an increase of learning rate in full-batch condition under the sum of squared errors and the mini-batch condition under the mean squared error. The apparent increase in learning rate and regularization effect can be attributed to the original input and additive noise in noisy copies, respectively. These results are confirmed in a numerical experiment in which we found that our result can be applied to usual off-line DA in an underparameterization scenario and can not in an over-parametrization scenario. Moreover, we experimentally investigated the training process of neural networks under DA with off-line noisy copies and found that our analysis on linear regression can be qualitatively applied to neural networks.

show abstract

Data Augmentation Approaches in Natural Language Processing: A Survey

Cited by 5 publications

References 78 publications

AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

Survey of Hallucination in Natural Language Generation

On Gradient Descent Training Under Data Augmentation with On-Line Noisy Copies

Contact Info

Product

Resources

About