ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

Abstract

Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1

Correcting diacritics and typos with a ByT5 transformer model

“…So the code point 353 of the letter "š" is translated into two bytes 197 and 161 while the letter "s" retains byte 115. [8] showed better results using a transformer model ByT5 at this byte-level tokens rather than on characters. Inspired by their success on transliteration and noisy text tasks, we also use the same byte-level tokenization.…”

“…Three years later, now there are plenty of similarly pre-trained publicly available models (e.g., at HuggingFace transformers library [94]). We also build our work on top of one such pre-trained ByT5 [8] model.…”

“…ByT5 [8] is a general-purpose pre-trained multilingual text-to-text model, based on earlier predecessor mT5 [100]. It completely disposes of SentencePiece [101] tokenizer, as it does not need any.…”

“…However, due to the higher "energy" (or "temperature") in the optimization the high η causes "bouncing" of parameter values and prevents settling in the best spot, resulting in the higher final training loss. An optimal learning rate value, as used during fine-tuning of the T5 family models [99,105,100,8] with Adafactor optimizer, is 0.001.…”

“…We used Adafactor [103] optimizer with a constant learning rate of 0.001. The same setup was employed by ByT5 [8] authors for fine-tuning experiments. Moreover, the Adafactor optimizer also has very little auxiliary storage compared to the other popular optimizer Adam [102].…”