DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks

Abstract

Data augmentation techniques have been widely used to improve machine learning performance as they enhance the generalization capability of models. In this work, to generate high quality synthetic data for low-resource tagging tasks, we propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings. For the supervised settings, we conduct extensive experiments on named entity recognition (NER), part of speech (POS) tagging and end-to-end target based sentiment analysis (E2E-TBSA) tasks. For the semi-supervised settings, we evaluate our method on the NER task under the conditions of given unlabeled data only and unlabeled data plus a knowledge base. The results show that our method can consistently outperform the baselines, particularly when the given gold training data are less. 1 * Equal contribution. Bosheng Ding and Linlin Liu are under the Joint PhD Program between Alibaba and Nanyang Technological University.

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

“…It then generates additional sentences that fit this label. Ding et al (2020) extend this idea for token level tasks.…”

MulDA: A Multilingual Data Augmentation Framework for Low-Resource Cross-Lingual NER

“…Although labeled sequence translation generates high quality multilingual NER training data, it adds limited variety since translation does not introduce new entities or contexts. Inspired by DAGA (Ding et al, 2020), we propose a generation-based multilingual data augmentation method to add more diversity to the training data. DAGA is a monolingual data augmentation method designed for sequence labeling tasks, which has been shown to be able to add significant diversity to the training data.…”

“…For NLP, back translation (Sennrich et al, 2016) is one of the most successful data augmentation approaches, which translates target-language monolingual data to the source language to generate more parallel data for MT model training. Other popular approaches include synonym replacement (Kobayashi, 2018), random deletion/swap/insertion Kumar et al, 2020), generation (Ding et al, 2020), etc. Data augmentation has also been proven to be useful in the cross-lingual settings Singh et al, 2020;Riabi et al, 2020;Qin et al, 2020;, but most of the exiting methods overlook the better utilization of multilingual training data when such resources are available.…”

“…However, except for some resource-rich languages (e.g., English, German), training sets for most of the other languages are still very limited. Moreover, it is usually expensive and time-consuming to annotate such data, particularly for low-resource languages (Kruengkrai et al, 2020). Therefore, zero-shot cross-lingual NER has attracted growing interest recently, especially with the influx of deep learning methods (Mayhew et al, 2017;Joty et al, 2017;Jain et al, 2019;.…”

Substructure Substitution: Structured Data Augmentation for NLP

“…Models trained on the augmented datasets are expected to be more robust against the considered noise. (Bergmanis et al, 2017;Liu et al, 2020a;Ding et al, 2020;Liu et al, 2020b, inter alia), or applying post-processing on the examples generated by pretrained models Wan et al, 2020;Yoo et al, 2020). In the data augmentation stage, given task-specific constraints, such models generate associated text accordingly.…”