Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

Abstract

Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus; MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.

MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

“…Multi-lingual language modeling Training language models in non-English languages has only recently received some attention. Kawakami et al (2017) evaluate on seven languages. Cotterell et al (2018) study 21 languages.…”

Are All Languages Equally Hard to Language-Model?

“…Recurrent neural language models can effectively learn complex dependencies, even in openvocabulary settings (Hwang and Sung, 2017;Kawakami et al, 2017). Whether the models are able to learn particular syntactic interactions is an intriguing question, and some methodologies have been presented to tease apart under what circumstances variously-trained models encode attested interactions (Linzen et al, 2016;Enguehard et al, 2017).…”

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

“…Multilingual Wikipedia Corpus The Multilingual Wikipedia Corpus (Kawakami, Dyer, and Blunsom 2017) contains 360 Wikipedia articles in English, French, Spanish, German, Russian, Czech, and Finnish. However, we re-tokenize the dataset, not only splitting on spaces (as Kawakami, Dyer, and Blunsom do) but also by splitting off each punctuation symbol as its own token.…”

“…Our model allows for both, namely e(w) and h i . Finally, we also compare against the character-aware model of Kawakami, Dyer, and Blunsom (2017), both without (HCLM) and with their additional cache (HCLMcache). To our knowledge, that model has the best previously known performance on the raw (i.e., open-vocab) version of the WikiText-2 dataset, but we see in both Table 1 and Table 2 that our model and the PURE-BPE baseline beat it.…”

“…Finally, the most relevant previous work is the (independently developed) model of Kawakami, Dyer, and Blunsom (2017), where each word has to be "spelled out" using a character-level RNN if it cannot be directly copied from the 45 1.386 1.36 1.317 1.45 1.414 1.42 1.362 1.88 1.856 1.70 1.652 1.63 1.598 FULL 1.45 1.387 1.36 1.319 1.51 1.465 1.42 1.363 1.95 1.928 1.79 1.751 1.74 1.709 Hwang and Sung (2017), there is no fixed vocabulary, so words that have fallen out of the cache have to be re-spelled. Our hierarchical generative story-specifically, the process that generates the lexicon-handles the re-use of words more gracefully.…”