mT5: A massively multilingual pre-trained text-to-text transformer

Abstract

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We describe the design and modified training of mT5 and demonstrate its stateof-the-art performance on many multilingual benchmarks. All of the code and model checkpoints used in this work are publicly available. 1

MT3: Multi-Task Multitrack Music Transcription

“…For example, T5 (Raffel et al, 2019) demonstrated that many language tasks previously addressed with separate models could be addressed using a single text-to-text encoder-decoder Transformer model. Extending this approach, mT5 (Xue et al, 2020) used a single Transformer to model multiple languages, demonstrating that a unified architecture could also serve as a general multilingual model, leveraging high-resource language datasets to improve model performance on lower-resource datasets.…”

“…In addition to removing the cumbersome task of constructing specialized architectures and loss functions for different instrumentations and datasets, our general output vocabulary also allows our model to be trained on a mixture of several datasets simultaneously, similar to how multilingual translation models such as mT5 are trained on several languages (Xue et al, 2020). This approach not only simplifies model design and training, but also increases the amount and diversity of training data available to the model.…”

“…In order to balance model performance on low-and high-resource datasets, we use a temperature sampling strategy for the mixing as follows: if dataset i has n i examples, we sample an example from that dataset with probability (n i / j n j ) 0.3 , similar to mT5 (Xue et al, 2020). This has the effect of increasing the frequency with which the model observes examples from low-resource datasets during training, while observing examples from high-resource datasets with lower frequency.…”

“…Unified framework for training: We define a tokenization scheme with a compact and flexible vocabulary to convert between model output tokens and multitrack MIDI files, enabling a sequence-tosequence approach inspired by Raffel et al (2019) and Xue et al (2020) that supports datasets with different combinations of instruments. This allows us to simultaneously leverage several datasets which were previously only used in isolation due to differences in instrumentation.…”

CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction

“…We decode the output by searching for occurrences of the predicted acronyms and long-forms and detecting their character spans in the input text. We use mT5 for our experiments (Xue et al 2021).…”