Dynamic Word Embeddings for Evolving Semantic Discovery

Abstract

Word evolution refers to the changing meanings and associations of words throughout time, as a byproduct of human language evolution. By studying word evolution, we can infer social trends and language constructs over different periods of human history. However, traditional techniques such as word representation learning do not adequately capture the evolving language structure and vocabulary. In this paper, we develop a dynamic statistical model to learn time-aware word vector representation. We propose a model that simultaneously learns time-aware embeddings and solves the resulting "alignment problem". This model is trained on a crawled NYTimes dataset. Additionally, we develop multiple intuitive evaluation strategies of temporal word embeddings. Our qualitative and quantitative tests indicate that our method not only reliably captures this evolution over time, but also consistently outperforms state-of-the-art temporal embedding approaches on both semantic accuracy and alignment quality.

Training Temporal Word Embeddings with a Compass

“…We compare TWEC with static models and with the state-of-the-art temporal models that have shown better performance according to the literature. We use the two main methodologies proposed to evaluate temporal embeddings so far: temporal analogical reasoning (Yao et al 2018) and held-out tests (Rudolph and Blei 2018). Our experiments can be easily replicated using the source code available online 1 .…”

“…Since they assume that the meaning of each word is fixed in time, they do not account for the semantic shifts of words. Thus, recent approaches have tried to capture the dynamics of language (Hamilton, Leskovec, and Jurafsky 2016;Bamler and Mandt 2018;Szymanski 2017;Yao et al 2018;Rudolph and Blei 2018).…”

Diachronic Cross-modal Embeddings

“…In Uricchio et al [29], the value of temporal information for the tasks of image annotation and retrieval, such as tag frequency, is recognised. In order to model the temporal behaviour of data, embeddings must retain temporal correlations [2,9,15,24,27,38]. The challenge resides in capturing such correlations and incorporating them in cross-modal embeddings.…”

“…Accordingly, textual descriptions of images reflect the way humans refer to each image at a given point in time, thus being susceptible to change over time, either due to the occurrence of external events [16] or simply by word meaning change across time [9,38]. Recently, word embedding models that capture language evolution over time (referred as distributional diachronic models) have been proposed [9,24,38]. In our setting we seek to obtain representations that capture the evolution of visual and textual correlations over time.…”

Learning Dynamic Author Representations with Temporal Language Models

“…The number of titles is not uniformly distributed, and grows quasi-exponentially with time: the year 1985 contains around 100 documents while the year 2017 has around 5K. -The New York Times [40] corpus (NYT) is composed of headlines from the New York Times newspaper spanning from 1990 to 2015, also split by years (26 timesteps). We also lower-cased the texts, but we use the NLTK [41] word tokenizer, and replaced every number with a special N token.…”