Multi-view Subword Regularization

Abstract

Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods (Kudo, 2018;Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms. 1

Specializing Multilingual Language Models: An Empirical Study

“…On the other hand, many adaptation techniques have focused on improving representation of the target language by modifying the model's vocabulary or tokenization schemes (Chung et al, 2020;Clark et al, 2021;Wang et al, 2021). This is wellmotivated: Artetxe et al (2020) emphasize representation in the vocabulary as a key factor for effective crosslingual transfer, while Rust et al (2021) find that MBERT's tokenization scheme for many languages is subpar.…”

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

“…Consequently, models such as multilingual BERT [Devlin et al, 2019] XLM-R [Conneau et al, 2020] employ the same subword tokenization algorithms as monolingual models, now applied to a massively multilingual corpus. In the multilingual setting, the problems of subword-based tokenization are exacerbated as tokens in languages with few data are over-segmented while high-frequency tokens are under-segmented, which limits cross-lingual transfer [Wang et al, 2021]. This motivates our work as well as recent work on character-level models.…”

“…To make a model more robust to morphological and compositional generalization, probabilistic segmentation algorithms such as subword regularization [Kudo, 2018] and BPE-dropout [Provilkov et al, 2020] have been proposed, which sample different segmentations during training. Recent methods propose to make models more robust for downstream tasks by enforcing prediction consistency between deterministic and probabilistic segmentations [Wang et al, 2021] and propose to update the tokenizer based on the downstream loss under different segmentations [Hiraoka et al, 2020[Hiraoka et al, , 2021. [He et al, 2020] proposed DPE (dynamic programming encoding), a segmentation-based tokenization algorithm based on dynamic programming.…”

“…As a result, models are brittle to rare words [Gong et al, 2018] and perturbations, both natural and adversarial [Belinkov and Bisk, 2018, Pruthi et al, 2019, Sun et al, 2020. In multilingual models, tokens in low-resource languages are split into many subwords, which impacts performance on those languages and deteriorates crosslingual transfer , Wang et al, 2021. Finally, a separate tokenization algorithm leads to a mismatch between the pre-training and downstream distribution of words when adapting pre-trained language models to new settings, which requires significant engineering effort to overcome.…”

Specializing Multilingual Language Models: An Empirical Study

“…On the other hand, many adaptation techniques have focused on improving representation of the target language by modifying the model's vocabulary or tokenization schemes (Chung et al, 2020;Clark et al, 2021;Wang et al, 2021). This is well-motivated: Artetxe et al (2020) emphasize representation in the vocabulary as a key factor for effective crosslingual transfer, while Rust et al (2020) find that MBERT's tokenization scheme for many languages is subpar.…”