Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Abstract

Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test. Word embeddings show promise as a diachronic tool, but have not been carefully evaluated. We develop a robust methodology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evolution. Using six historical corpora spanning four languages and two centuries, we propose two quantitative laws of semantic change: (i) the law of conformity-the rate of semantic change scales with an inverse power-law of word frequency; (ii) the law of innovation-independent of frequency, words that are more polysemous have higher rates of semantic change.

Exploring Diachronic Lexical Semantics with JeSemE

“…In contrast to other corpus exploration tools, JESEME is based on cutting-edge word embedding technology (Levy et al, 2015;Hamilton et al, 2016;Hahn, 2016a, 2017) and provides access to five popular corpora for the English and German language. JESEME is also the first tool of its kind and under continuous development.…”

“…Future technical work will add functionality to compare words across corpora which might require a mapping between embeddings (Kulkarni et al, 2015;Hamilton et al, 2016) and provide optional stemming routines. Both goals come with an increase in precomputed similarity values and will thus necessitate storage optimizations to ensure long-term availability.…”

“…We follow Kim et al (2014) in choosing such a visualization, while we refrain from using the two-dimensional projection used in other studies (Kulkarni et al, 2015;Hamilton et al, 2016). We stipulate that the latter could Figure 2: Screenshot of JESEME's result page when searching for the lexical item "heart" in COHA.…”

Neoveille, a Web Platform for Neologism Tracking

“…Applied to semantic change, if you have at your disposal a bunch of diachronic corpora, you can build the semantic vectors of any lexical unit corresponding to several periods, and track the changes from one period to another. First experiments have been proposed by (Hamilton et al, 2016). The main advantage of this approach resides in the fact that it proposes for a given word a list of semantically similar words, among which synonyms and hypernyms, which permits to clearly explicit the meaning of a word.…”

Exploring Temporal Analysis of Tweet Content from Cultural Events

“…This is problematic when these models are used as input for information retrieval tasks, such as automatic event extraction, automatic summarization, etc. Based on this observation we propose a preliminary work close to those initiated in [1,7] that seeks to compare two word embedding-based models: one ignoring the temporal aspect of the messages, using the state-of-the-art Word2Vec model [9], and one taking advantage of the emission date of tweets, using the Temporal Embedding approach.…”