Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora

Abstract

The problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and-as we show in this work-result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew).

TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora

“…More broadly, prior work by Wendlandt et al (2018), Antoniak andMimno (2018), andGonen et al (2020), among others, has also shown embedding stability to be a concern in models trained on larger corpora than those used in this work. However, the role of random embedding effects on previous qualitative studies using word embeddings (e.g., Kulkarni et al (2015), Hamilton et al (2016)) has not been evaluated.…”

“…This method also avoids the conceptual difficulties and low replicability of comparing embedding spaces numerically (e.g. by cosine distances) (Gonen et al, 2020). However, even nearest neighborhoods are often unstable, and vary dramatically across runs of the same embedding algorithm on the same corpus (Wendlandt et al, 2018;Antoniak and Mimno, 2018).…”

“…However, quantitative, vector-based comparison of embedding spaces faces significant conceptual challenges, such as a lack of appropriate alignment objectives and empirical instability (Gonen et al, 2020). While nearest neighbor-based change measurement has been proposed (Newman-Griffis and Fosler-Lussier, 2019;Gonen et al, 2020), its efficacy for small corpora with limited vocabularies remains to be determined. Our novel embedding confidence measure offers a step in this direction (see §6.3 for further discussion), but further research is needed.…”

Words with Consistent Diachronic Usage Patterns are Learned Earlier: A Computational Analysis Using Temporally Aligned Word Embeddings

“…This approach builds on recent advancements in the study of language change using distributed temporal word embeddings (Bianchi et al., 2020; Di Carlo et al., 2019; Hilpert & Perek, 2015; Kim et al., 2014; Kulkarni et al., 2015; Perek, 2014; Sagi et al., 2011; Yao et al., 2018). Finally, we implemented and evaluated three different ways of capturing language change, which target both item‐ and neighborhood‐level patterns of change in usage patterns (Gonen et al., 2020; Hamilton et al., 2016a), offering a more thorough and insightful characterization of the phenomenon at hand.…”

“…We use three different methods to quantify language change 9 (Gonen et al., 2020; Hamilton et al., 2016a). In order to ensure comparability across words, we only computed measures of language change for words that appeared at least 25 times in each of the six slices in which the corpus was divided, such that reliable lexical representations could be learned.…”