The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings

Abstract

We introduce 'POLAR' -a framework that adds interpretability to pre-trained word embeddings via the adoption of semantic differentials. Semantic differentials are a psychometric construct for measuring the semantics of a word by analysing its position on a scale between two polar opposites (e.g., cold -hot, soft -hard). The core idea of our approach is to transform existing, pre-trained word embeddings via semantic differentials to a new "polar" space with interpretable dimensions defined by such polar opposites. Our framework also allows for selecting the most discriminative dimensions from a set of polar dimensions provided by an oracle, i.e., an external source. We demonstrate the effectiveness of our framework by deploying it to various downstream tasks, in which our interpretable word embeddings achieve a performance that is comparable to the original word embeddings. We also show that the interpretable dimensions selected by our framework align with human judgement. Together, these results demonstrate that interpretability can be added to word embeddings without compromising performance. Our work is relevant for researchers and engineers interested in interpreting pre-trained word embeddings. CCS CONCEPTS• Computing methodologies → Machine learning approaches. KEYWORDS word embeddings, neural networks, interpretable, semantic differential ACM Reference Format:

Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model

“…We develop our computational SCM using labelled data from Nicolas et al (2020) and the POLAR framework for interpretable word embeddings (Mathew et al, 2020), and then apply it to stereotype and anti-stereotype data from StereoSet (Nadeem et al, 2020). Details are provided in the following sections.…”

“…Rather than using an unsupervised approach such as PCA, we choose the POLAR framework introduced by Mathew et al (2020). This framework seeks to improve the interpretability of word embeddings by leveraging the concept of 'semantic differentials,' a psychological rating scale which contrasts bipolar adjectives, e.g.…”

“…In contrast to the standard POLAR framework introduced by Mathew et al (2020), we do not have a set of polar opposite word pairs, each representing a different interpretable dimension, but rather a set of words for each of the concepts warmth, coldness, competence, and incompetence from Nicolas et al (2020). Therefore, we use a slightly different formulation to obtain the polar directions associated with warmth and competence.…”

“…We consider here only noncontextual word embeddings, in line withMathew et al (2020). Because the POLAR framework is based on linear algebraic computations, it is not immediately obvious whether it will extend directly to contextualized embeddings, which are notably anisotropic(Ethayarajh, 2019).…”

Interpretable contrastive word mover's embedding

“…In Arora et al (2018) the authors proposed a linear algebraic structure to explain the polysemy of words. Recent works attempted to explain the meaning of each dimension, such as the sparse word embedding Faruqui et al (2015); Panigrahi et al (2019) and the POLAR Framework Mathew et al (2020). To make WMD embeddings interpretable, Xu et al (2018) proposed an unsupervised topic model in the representation space.…”