Morphological and Language-Agnostic Word Segmentation for NMT

Macháček, Dominik; Vidra, Jonáš; Bojar, Ondřej

doi:10.1007/978-3-030-00794-2_30

Cited by 10 publications

(9 citation statements)

References 5 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Banerjee and Bhattacharyya (2018) also use unsupervised morphological units generated by Morfessor (Virpioja et al, 2013) as input for a neural machine translation system and report improvements for low-resource conditions. Macháček et al (2018) follow a similar approach for translation into Czech on WMT data, but were not able to obtains improvements over the standard BPE approach.…”

Section: Related Workmentioning

confidence: 99%

A Statistical Extension of Byte-Pair Encoding

Vilar¹,

Federico²

2021

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

View full text Add to dashboard Cite

Sub-word segmentation is currently a standard tool for training neural machine translation (MT) systems and other NLP tasks. The goal is to split words (both in the source and target languages) into smaller units which then constitute the input and output vocabularies of the MT system. The aim of reducing the size of the input and output vocabularies is to increase the generalization capabilities of the translation model, enabling the system to translate and generate infrequent and new (unseen) words at inference time by combining previously seen sub-word units. Ideally, we would expect the created units to have some linguistic meaning, so that words are created in a compositional way. However, the most popular wordsplitting method, Byte-Pair Encoding (BPE), which originates from the data compression literature, does not include explicit criteria to favor linguistic splittings, nor to find the optimal sub-word granularity for the given training data. In this paper, we propose a statistically motivated extension of the BPE algorithm and an effective convergence criterion that avoids the costly experimentation cycle needed to select the best sub-word vocabulary size. Experimental results with morphologically rich languages show that our model achieves nearly-optimal BLEU scores and produces morphologically better word segmentations, which allows to outperform BPE's generalization in the translation of sentences containing new words, as shown via human evaluation.

show abstract

Section: Related Workmentioning

confidence: 99%

A Statistical Extension of Byte-Pair Encoding

Vilar¹,

Federico²

2021

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

View full text Add to dashboard Cite

show abstract

“…On the other side of the spectrum, it has been observed that automatically learned subwords generally do not correspond to linguistic entities such as morphemes, suffixes, affixes etc. However, linguistically-motivated subword units as proposed by Huck et al (2017), Macháček et al (2018), Ataman et al (2017), Pinnis et al (2017) that also take morpheme boundaries into account do not always improve over completely data-driven ones.…”

Section: Subword-unit-based Nmtmentioning

confidence: 99%

Neural Machine Translation: A Review

Stahlberg¹

2020

jair

204

View full text Add to dashboard Cite

The field of machine translation (MT), the automatic translation of written text from one natural language into another, has experienced a major paradigm shift in recent years. Statistical MT, which mainly relies on various count-based models and which used to dominate MT research for decades, has largely been superseded by neural machine translation (NMT), which tackles translation with a single neural network. In this work we will trace back the origins of modern NMT architectures to word and sentence embeddings and earlier examples of the encoder-decoder network family. We will conclude with a short survey of more recent trends in the field.

show abstract

“…Using unsupervisedly obtained "morphological" subwords on the other hand, only Ataman and Federico (2018b) find that a model based on Morfessor FlatCat can outperform BPE; Zhou (2018), , Macháček et al (2018), and Saleva and Lignos (2021) find no reliable improvement over BPE for translation. Banerjee and Bhattacharyya (2018) analyze translations obtained segmenting with Morfessor and BPE, and conclude that a possible improvement depends on the similarity of the languages.…”

Section: Comparing Morphological Segmentation To Bpe and Friendsmentioning

confidence: 99%

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Mielke,

Alyafeai,

Salesky

et al. 2021

Preprint

View full text Add to dashboard Cite

What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural language processing (NLP) models operated over words, treating those as discrete and atomic tokens, but starting with byte-pair encoding (BPE), subword-based approaches have become dominant in many areas, enabling small vocabularies while still allowing for fast inference. Is the end of the road character-level model or byte-level processing? In this survey, we connect several lines of work from the pre-neural and neural era, by showing how hybrid approaches of words and characters as well as subwordbased approaches based on learned segmentation have been proposed and evaluated. We conclude that there is and likely will never be a silver bullet singular solution for all applications and that thinking seriously about tokenization remains important for many applications. * Working group chairs 1 A Twitter bot with (human-curated) outputs of a language model based on GPT2 (Radford et al., 2019) and trained on tweets of Twitter poet @dril; https://twitter.com/ dril_gpt2/status/1373596260612067333.

show abstract

Morphological and Language-Agnostic Word Segmentation for NMT

Cited by 10 publications

References 5 publications

A Statistical Extension of Byte-Pair Encoding

A Statistical Extension of Byte-Pair Encoding

Neural Machine Translation: A Review

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Contact Info

Product

Resources

About