Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue

Abstract

Scaling dialogue systems to a multitude of domains, tasks and languages relies on costly and time-consuming data annotation for different domain-task-language configurations. The annotation efforts might be substantially reduced by the methods that generalise well in zero-and few-shot scenarios, and also effectively leverage external unannotated data sources (e.g., Web-scale corpora). We propose two methods to this aim, offering improved dialogue natural language understanding (NLU) across multiple languages: 1) Multi-SentAugment, and 2) LayerAgg. Multi-SentAugment is a self-training method which augments available (typically few-shot) training data with similar (automatically labelled) in-domain sentences from large monolingual Web-scale corpora. LayerAgg learns to select and combine useful semantic information scattered across different layers of a Transformer model (e.g., mBERT); it is especially suited for zero-shot scenarios as semantically richer representations should strengthen the model's cross-lingual capabilities. Applying the two methods with state-of-the-art NLU models obtains consistent improvements across two standard multilingual NLU datasets covering 16 diverse languages. The gains are observed in zero-shot, few-shot, and even in full-data scenarios. The results also suggest that the two methods achieve a synergistic effect: the best overall performance in few-shot setups is attained when the methods are used together.

Transfer-Free Data-Efficient Multilingual Slot Labeling

“…The goal in Step 1 is to ensure high recall (i.e., to avoid too aggressive filtering), which is why we opt for the earlier stopping. As a baseline, we fine-tune XLM-R for the token classification task, as the standard SL task format (Xu et al, 2020;Razumovskaia et al, 2022b). Detailed training hyperparameters are provided in Appendix B.…”

“…The current approach to mitigate the issue is the standard cross-lingual transfer. The main 'transfer' assumption is that a suitable large English annotated dataset is always available for a particular task and domain: (i) the systems are then trained on the English data and then directly deployed to the target language (i.e., zero-shot transfer), or (ii) further adapted to the target language relying on a small set of target language examples (Xu et al, 2020;Razumovskaia et al, 2022b) which are combined with the large English dataset (i.e., few-shot transfer). However, this assumption might often be unrealistic in the context of TOD due to a large number of potential tasks and domains that should be supported by TOD systems .…”

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

“…Firstly, several automated approaches to reduce human effort were proposed, predominantly focusing on data augmentation. These approaches, previously applied to multilingual dialogue, include (i) word or span substitution, creating code-switched data between source and target languages Krishnan et al, 2021;Qin et al, 2021) or synonymous span substitution in a target language (Louvan & Magnini, 2020b)2020; and (ii) semi-supervised training with target language sentence retrieval (Razumovskaia et al, 2022).…”

SILVER: Self Data Augmentation for Out-of-Scope Detection in Dialogues

“…Due to the lack of OOS annotations in openworld settings, previous research usually detects OOS samples indirectly such as resorting to inscope (INS) samples. Recently, data augmentation methods (Ng et al, 2020;Razumovskaia et al, 2022) have made it possible to detect OOS directly using a binary classifier.…”