Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

Abstract

We show how the spellings of known words can help us deal with unknown words in open-vocabulary NLP tasks. The method we propose can be used to extend any closedvocabulary generative model, but in this paper we specifically consider the case of neural language modeling. Our Bayesian generative story combines a standard RNN language model (generating the word tokens in each sentence) with an RNNbased spelling model (generating the letters in each word type). These two RNNs respectively capture sentence structure and word structure, and are kept separate as in linguistics. By invoking the second RNN to generate spellings for novel words in context, we obtain an open-vocabulary language model. For known words, embeddings are naturally inferred by combining evidence from type spelling and token context. Comparing to baselines (including a novel strong baseline), we beat previous work and establish state-of-the-art results on multiple datasets. prior on embeddings · p spell (σ(w) | e(w)) spelling model for all types lexicon generation · n ∏ i=1 p LM (w i | w <i , e) lexeme-level recurrent language model for all tokens 1 Before extension to the open-vocabulary case, found in Eq. (3).

WordTokenizers.jl: Basic tools for tokenizing natural language in Julia

“…With various lexers written for the TokenBuffer API, users can also create their high-speed custom tokenizers with ease. The package also provides a simple reversible tokenizer (Mielke, 2019;Mielke & Eisner, 2018) that works by leaving certain merge symbols, as a means to reconstruct tokens into the original string.…”

Languages with more speakers tend to be harder to (machine-)learn

“…The language-specific value of this random slope parameter is then indicative of a stronger or weaker relationship between H and f for this language. As information encoding units, we estimate on two levels: on the level of words and, instead of estimating on the level of characters, we tokenize our text into sub-word units by byte pair encoding (BPE) 63,64 which plays an important role in many state-of-the-art natural language model applications 65,66 and provides strong baseline results on a multilingual corpus 67 . In total, we trained seven different LMs on the dataranging from very simple n-gram models to stateof-the-art deep neural networks (Table 3).…”

Human languages trade off complexity against efficiency

“…Some of the contexts are allowed to be non-contiguous in order to capture longer-term dependencies 48 and CMIX uses long short-term memory 61 (LSTM) trained by backpropagation as a byte-level mixer 59 . In addition, instead of estimating on the level of either characters or words, we tokenize our text into sub-word units by byte pair encoding (BPE) 27,62 which plays an important role in many state-of-the-art natural language model applications such as GPT-3 63 or SentencePiece 64 and provides strong baseline results on a multilingual corpus 65 .…”