A Formal Perspective on Byte-Pair Encoding

Zouhar, Vilém; Meister, Clara; Gastaldi, Juan Luis; Du, Lan; Vieira, Tim; Sachan, Mrinmaya; Cotterell, Ryan

doi:10.18653/v1/2023.findings-acl.38

Cited by 3 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This procedure is repeated until we fill the predefined vocabulary budget. Zouhar et al (2023) show that this greedy approach is approximately optimal when searching for the vocabulary (merge sequence). We are however interested in intentionally suboptimal vocabularies.…”

Section: E1 Byte-pair Encodingmentioning

confidence: 99%

Tokenization and the Noiseless Channel

Zouhar¹,

Meister²,

Gastaldi³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to efficient channel usage, where the channel is the means by which some input is conveyed to the model and efficiency can be quantified in information-theoretic terms as the ratio of the Shannon entropy to the maximum possible entropy of the token distribution. Yet, an optimal encoding according to Shannon entropy assigns extremely long codes to low-frequency tokens and very short codes to high-frequency tokens. Defining efficiency in terms of Rényi entropy, on the other hand, penalizes distributions with either very high or very low-frequency tokens. In machine translation, we find that across multiple tokenizers, the Rényi entropy with α = 2.5 has a very strong correlation with BLEU: 0.78 in comparison to just −0.32 for compressed length.

show abstract

Section: E1 Byte-pair Encodingmentioning

confidence: 99%

Tokenization and the Noiseless Channel

Zouhar¹,

Meister²,

Gastaldi³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

Investigation and Implementation of AI-HDLCoder for Automated VHDL Code Synthesis and Code Generation for Hardware SoC Development

Romashchenko,

Brutscheck,

Chmielewski

2024

2024 35th Irish Signals and Systems Conference (ISSC)

View full text Add to dashboard Cite

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Manohar,

A R,

Rajan

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

This article presents the research work on improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling. The speech recognition system is built using a deep neural network–hidden Markov model (DNN-HMM)-based automatic speech recognition (ASR). We propose a novel method, syllable-byte pair encoding (S-BPE), that combines linguistically informed syllable tokenization with the data-driven tokenization method of byte pair encoding (BPE). The proposed method ensures words are always segmented at valid pronunciation boundaries. On a text corpus that has been divided into tokens using the proposed method, we construct statistical n-gram language models and assess the modeling effectiveness in terms of both information-theoretic and corpus linguistic metrics. A comparative study of the proposed method with other data-driven (BPE, Morfessor, and Unigram), linguistic (Syllable), and baseline (Word) tokenization algorithms is also presented. Pronunciation lexicons of subword tokenized units are built with pronunciation described as graphemes. We develop ASR systems employing the subword tokenized language models and pronunciation lexicons. The resulting ASR models are comprehensively evaluated to answer the research questions regarding the impact of subword tokenization algorithms on language modeling complexity and on ASR performance. Our study highlights the strong performance of the hybrid S-BPE tokens, achieving a notable 10.6% word error rate (WER), which represents a substantial 16.8% improvement over the baseline word-level ASR system. The ablation study has revealed that the performance of S-BPE segmentation, which initially underperformed compared to syllable tokens with lower amounts of textual data for language modeling, exhibited steady improvement with the increase in LM training data. The extensive ablation study indicates that there is a limited advantage in raising the n-gram order of the language model beyond $$n=3$$ n = 3 . Such an increase results in considerable model size growth without significant improvements in WER. The implementation of the algorithm and all associated experiments are available under an open license, allowing for reproduction, adaptation, and reuse.

show abstract

A Formal Perspective on Byte-Pair Encoding

Cited by 3 publications

References 0 publications

Tokenization and the Noiseless Channel

Tokenization and the Noiseless Channel

Investigation and Implementation of AI-HDLCoder for Automated VHDL Code Synthesis and Code Generation for Hardware SoC Development

Improving speech recognition systems for the morphologically complex Malayalam language using subword tokens for language modeling

Contact Info

Product

Resources

About