Adversarial Subword Regularization for Robust Neural Machine Translation

Park, Jung Soo; Sung, Mujeen; Lee, Jinhyuk; Kang, Jaewoo

doi:10.18653/v1/2020.findings-emnlp.175

Cited by 4 publications

(4 citation statements)

References 30 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The replacement operation of m-th token x m to the arbitrary token x can be written as δ(x m , x) := e(x) − e(x m ), where e(•) denotes embedding look-up. We induce a virtual adversarial token by the following criteria (Ebrahimi et al, 2017;Michel et al, 2019;Cheng et al, 2019;Wallace et al, 2019;Park et al, 2020):…”

Section: Gradient Informationmentioning

confidence: 99%

Consistency Training with Virtual Adversarial Discrete Perturbation

Park¹,

Kim²,

Kang³

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark 1 .

show abstract

Section: Gradient Informationmentioning

confidence: 99%

Consistency Training with Virtual Adversarial Discrete Perturbation

Park¹,

Kim²,

Kang³

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

show abstract

“…Works involving Input Perturbation Apart from the works mentioned above, some works introduce subword uncertainty at the subword segmentation stage, including sampling multiple subword candidates (Kudo, 2018), applying subword dropout (Park et al, 2020) or producing adversarial subword segmentation (Provilkov et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Prediction Difference Regularization against Perturbation for Neural Machine Translation

Guo¹,

Ma²,

Zhang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Regularization methods applying input perturbation have drawn considerable attention and have been frequently explored for NMT tasks in recent years. Despite their simplicity and effectiveness, we argue that these methods are limited by the under-fitting of training data. In this paper, we utilize prediction difference for ground-truth tokens to analyze the fitting of token-level samples and find that underfitting is almost as common as over-fitting. We introduce prediction difference regularization (PD-R), a simple and effective method that can reduce over-fitting and under-fitting at the same time. For all token-level samples, PD-R minimizes the prediction difference between the original pass and the input-perturbed pass, making the model less sensitive to small input changes, thus more robust to both perturbations and under-fitted training data. Experiments on three widely used WMT translation tasks show that our approach can significantly improve over existing perturbation regularization methods. On WMT16 En-De task, our model achieves 1.80 SacreBLEU improvement over vanilla transformer.

show abstract

“…To tokenize words at the morpheme level, we utilize KoNLPy, an open-source library for Korean text that provides a number of different tokenizers with different parsing rules and methods. In the training process, we augmented two types of tokenized sentences from each sentence in Korean text with two different tokenizers, Mecab and Komoran, as illustrated in 10% on average, but also has the effect of subword regularization (Kudo, 2018;Park et al, 2020a). Accordingly, our model utilizes various sets of subtoken candidates, that yield robustness to typos or slangs.…”

Section: Datasetmentioning

confidence: 99%

KOAS: Korean Text Offensiveness Analysis System

Park¹,

Kim²,

Cho³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

View full text Add to dashboard Cite

Warning: This manuscript contains a certain level of offensive expression.As communication through social media platforms has grown immensely, the increasing prevalence of offensive language online has become a critical problem. Notably in Korea, one of the countries with the highest Internet usage, automatic detection of offensive expressions has recently been brought to attention. However, morphological richness and complex syntax of Korean causes difficulties in neural model training. Furthermore, most of previous studies mainly focus on the detection of abusive language, disregarding implicit offensiveness and underestimating a different degree of intensity. To tackle these problems, we present KOAS, a system that fully exploits both contextual and linguistic features and estimates an offensiveness score for a text. We carefully designed KOAS with a multi-task learning framework and constructed a Korean dataset for offensive analysis from various domains. Refer for a detailed demonstration. 1

show abstract

Adversarial Subword Regularization for Robust Neural Machine Translation

Cited by 4 publications

References 30 publications

Consistency Training with Virtual Adversarial Discrete Perturbation

Consistency Training with Virtual Adversarial Discrete Perturbation

Prediction Difference Regularization against Perturbation for Neural Machine Translation

KOAS: Korean Text Offensiveness Analysis System

Contact Info

Product

Resources

About