Understanding the stability of medical concept embeddings

Abstract

Frequency is one of the major factors for training quality word embeddings. Several studies have recently discussed the stability of word embeddings in general domain and suggested factors influencing the stability. In this work, we conduct a detailed analysis on the stability of concept embeddings in medical domain, particularly in relations with concept frequency. The analysis reveals the surprising high stability of low‐frequency concepts: low‐frequency (<100) concepts have the same high stability as high‐frequency (>1,000) concepts. To develop a deeper understanding of this finding, we propose a new factor, the noisiness of context words, which influences the stability of medical concept embeddings regardless of high or low frequency. We evaluate the proposed factor by showing the linear correlation with the stability of medical concept embeddings. The correlations are clear and consistent with various groups of medical concepts. Based on the linear relations, we make suggestions on ways to adjust the noisiness of context words for the improvement of stability. Finally, we demonstrate that the linear relation of the proposed factor extends to the word embedding stability in general domain.

Are Word Embedding Methods Stable and Should We Care About It?

“…Word2Vec [8,11,21,28,32], GloVe [1,22,32], PPMI [1,22,32], SGNS [1,18,22], SVD [22], LSA [1], SGHS [18] Word2Vec, GloVe and fastText…”

“…• Wikipedia (1.5B) [22] • NYT(58M) and Europarl (61M) [32] • Brown, Project Gutenberg and Reuters (10k each) [8] • US Federal Courts of Appeals (38k), NYT (22k) and Reddit (26k) [1] • NIPS between 2007 and 2012 (2M) [11] • Ohsumed dataset (34M) [21] • Google N-gram cor-pus: English Fiction(4.8B) and German(0.7B) [18] • BNC and ACL An-thology Reference corpus [28] Wikipedia, News-Crawl (2007), Lyrics and Europarl (50M each)…”

“…• Medium frequency words have the lowest reliability [18] • Frequency is not a major factor in stability [32] • Words having a low and high frequency range have a tendency to display more variation. Medium frequency words show more stability [28] • Frequency does not directly affect the stability of medical word embeddings [21] •All word groups(low, medium, and high frequency) show high variance in word stability.…”

“…Stability evaluation of WEMs has received a lot of attention in the recent past [1,8,11,21,22,32]. Such evaluation is crucial because word embeddings are widely used today for a variety of NLP tasks such as sentiment analysis [23,30], named entity recognition [20,29] and part-of-speech (POS) tagging [31,32].…”