Methods for Numeracy-Preserving Word Embeddings

Abstract

Word embedding models are typically able to capture the semantics of words via the distributional hypothesis, but fail to capture the numerical properties of numbers that appear in a text. This leads to problems with numerical reasoning involving tasks such as question answering. We propose a new methodology to assign and learn embeddings for numbers. Our approach creates Deterministic, Independentof-Corpus Embeddings (referred to as DICE) for numbers, such that their cosine similarity reflects the actual distance on the number line. DICE outperforms a wide range of pre-trained word embedding models across multiple examples of two tasks: (i) evaluating the ability to capture numeration and magnitude; and (ii) to perform list maximum, decoding, and addition. We further explore the utility of these embeddings in downstream applications by initializing numbers with our approach for the task of magnitude prediction. We also introduce a regularization approach to learn model-based embeddings of numbers in a contextual setting.

Representing Numbers in NLP: a Survey and a Vision

“…DICE Determinisitic Independent-of-Corpus Embeddings (Sundararaman et al, 2020) is an attempt to handcraft number encoder 3 f so as to preserve the relative magnitude between two numerals and their embeddings. Given two scalars i and j, and their embeddings f (i) and f (j), the cosine distance between f (i) and f (j) is intended to monotonically increase/decrease with the Euclidean distance between i and j.…”

“…Some progress has already been made in this transfer learning setup, e.g., GenBERT (Geva et al, 2020), finetuned on a synthetic dataset of arithmetic problems, is found to score higher on DROP QA. Similarly, DICE (Sundararaman et al, 2020), optimized for numeration, improves score on Numeracy600K order-of-magnitude prediction task. Going forward, we need several such studies, ideally for each pair of tasks to see whether some numeracy skills help models generalize to others.…”

“…Direction: Some proposed methods are encoderonly, e.g., DICE (Sundararaman et al, 2020), while some can be decoder-only, e.g., those requiring sampling from a parameterized distribution (Spokoyny and Berg-Kirkpatrick, 2020).…”

“…Model designers must also make a choice on coverage: whether to target a broad or a narrow range of numbers to be represented. Multi-class classification (Zhang et al, 2020) over a fixed number of bins, restricts the range of numbers expressed, as do DICE embeddings (Sundararaman et al, 2020). Value embeddings are continuous and theoretically unrestricted, but must practically be capped for bugfree training.…”

Mining Numbers in Text: A Survey

“…They find "prototype numbers" by clustering, and represent numbers as a weighted average of these prototypes. Sundararaman et al [21] proposed to learn embeddings for numbers, which reflect the distance of two numbers in the number line, independently from words. Table 6 summarizes these previous methods.…”