Temporal Word Analogies: Identifying Lexical Replacement with Diachronic Word Embeddings

Abstract

This paper introduces the concept of temporal word analogies: pairs of words which occupy the same semantic space at different points in time. One well-known property of word embeddings is that they are able to effectively model traditional word analogies ("word w 1 is to word w 2 as word w 3 is to word w 4 ") through vector addition. Here, I show that temporal word analogies ("word w 1 at time t α is like word w 2 at time t β ") can effectively be modeled with diachronic word embeddings, provided that the independent embedding spaces from each time period are appropriately transformed into a common vector space. When applied to a diachronic corpus of news articles, this method is able to identify temporal word analogies such as "Ronald Reagan in 1987 is like Bill Clinton in 1997", or "Walkman in 1987 is like iPod in 2007". BackgroundThe meanings of utterances change over time, due both to changes within the linguistic system and to changes in the state of the world. For example, the meaning of the word awful has changed over the past few centuries from something like "aweinspiring" to something more like "very bad", due to a process of semantic drift. On the other hand, the phrase president of the United States has meant different things at different points in time due to the fact that different people have occupied that same position at different times. These are very different types of changes, and the latter may not even be considered a linguistic phenomenon, but both types of change are relevant to the concept of temporal word analogies.I define a temporal word analogy (TWA) as a pair of words which occupy a similar semantic space at different points in time. For example, assuming that there is a semantic space associated with "President of the USA", this space was occupied by Ronald Reagan in the 1980s, and by Bill Clinton in the 1990s. So a temporal analogy holds: "Ronald Reagan in 1987 is like Bill Clinton in 1997".Distributional semantics methods, particularly vector-space models of word meanings, have been employed to study both semantic change and word analogies, and as such are well-suited for the task of identifying TWAs. The principle behind these models, that the meaning of words can be captured by looking at the contexts in which they appear (i.e. other words), is not a recent idea, and is generally attributed to Harris (1954) or Firth (1957. The modern era of applying this principle algorithmically began with latent semantic analysis (LSA) (Landauer and Dumais, 1997), and the recent explosion in popularity of word embeddings is largely due to the very effective word2vec neural network approach to computing word embeddings (Mikolov et al., 2013a). In these types of vector space models (VSMs), the meaning of a word is represented as a multi-dimensional vector, and semantically-related words tend to have vectors that relate to one another in regular ways (e.g. by occupying nearby points in the vector space). One factor in word embeddings' recent popularity is their eye-catching ability to m...

Mapping the physics research space: a machine learning approach

“…to understand how scientists or research topics move over time. Indeed, in the NLP literature, some approaches have been proposed to study how word analogies and semantic meaning change over time (see for example [90,91]). Similar techniques could be used in our context to study the temporal evolution of science.…”

Diachronic degradation of language models: Insights from social media

“…The language of social media posts can be used to study semantic drift over short periods of time, even from a dataset of 554 social media users. These methods can also find application in the study of other linguistic phenomena such as polysemy (Hamilton et al, 2016b;Szymanski, 2017). However, there is a need to disentangle which differences are due to the changing use of language from the ones due to changes in topics and trends on social media.…”

Training Temporal Word Embeddings with a Compass

“…Pairwise alignment-based approaches align pairs of vector spaces to a unique coordinate system: Kim et al and Tredici, Nissim, and Zaninello align consecutive temporal vectors through neural network initialization; other authors apply various linear transformations after training that minimize the distance between the pairs of vectors associated with each word in two vector spaces (Kulkarni et al 2015;Hamilton, Leskovec, and Jurafsky 2016;Szymanski 2017;Zhang et al 2016).…”

“…Since they assume that the meaning of each word is fixed in time, they do not account for the semantic shifts of words. Thus, recent approaches have tried to capture the dynamics of language (Hamilton, Leskovec, and Jurafsky 2016;Bamler and Mandt 2018;Szymanski 2017;Yao et al 2018;Rudolph and Blei 2018).…”

“…This approach is well motivated but may lead, for some words, to excessively smoothen differences between meanings that have shifted along time. A remarkable limitation of current TWEMs is related to the assumptions they make on the size of the corpus needed for training: while some methods like (Szymanski 2017;Hamilton, Leskovec, and Jurafsky 2016) require a huge amount of training data, which may be difficult to acquire in several application domains, other methods like (Yao et al 2018;Rudolph and Blei 2018) may not scale well when trained with big datasets.…”