On the Sentence Embeddings from Pre-trained Language Models

Abstract

Pre-trained contextual representations like BERT have achieved great success in natural language processing. However, the sentence embeddings from the pre-trained language models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We first reveal the theoretical connection between the masked language model pre-training objective and the semantic similarity task theoretically, and then analyze the BERT sentence embeddings empirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks. The code is available at https://github.com/ bohanli/BERT-flow. * The work was done when BL was an intern at ByteDance.

Positional Artefacts Propagate Through Masked Language Model Embeddings

“…They relate this observation to the Zipfian nature of word distributions, where the vast majority of words are infrequent. Li et al (2020a) extend this insight specifically to BERT and show that, while high frequency words concentrate densely, low frequency words are much more sparsely distributed. Though we do not attempt to dispute these claims with our findings, we do hope our experiments will highlight the important role that positional embeddings play in the representational geometry of Transformer-based models.…”

“…When cosine similarity is viewed primarily as means of semantic comparison between word or sentence vectors, the prospect of calculating cosine similarity for a benchmark like WiC or STS-B becomes erroneous. Though an examination of distance metrics is outside the scope of this study, we acknowledge similar points as having been addressed in regards to static word embeddings (Mimno and Thompson, 2017) as well as contextualized ones (Li et al, 2020b). Likewise, we would like to stress that our manual clipping operation was performed for illustrative purposes and that interested researchers should employ more systematic post-hoc normalization strategies, e.g.…”

“…Nonetheless, much of this work likewise fails to engage with the raw vector spaces of language models, preferring instead to focus its analysis on the transformed vectors. Indeed, the fraction of work that has done the former has shed some curious insights: that untransformed BERT sentence representations still lag behind word embeddings across a variety of semantic benchmarks (Reimers and Gurevych, 2019) and that the vector spaces of language models are explicitly anisotropic (Ethayarajh, 2019;Li et al, 2020a). Certainly, an awareness of the patterns inherent to models' untransformed vector spaces -even if shallow -can only benefit the transformation-based analyses outlined above.…”

ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling

“…However, the sentence or document embeddings derived from such an MLM without finetuning on in-domain data is shown to be inferior in terms of the ability to capture semantic information that can be used in similarity related tasks (Reimers and Gurevych, 2019). Instead of using the [CLS] vector to obtain sentence embeddings, in this paper we take the average of context embeddings from last two layers as these are shown to be consistently better than using [CLS] vector (Reimers and Gurevych, 2019;Li et al, 2020).…”

Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models

“…Recent years have seen huge success of pretrained language models across a wide range of NLP tasks (Devlin et al, 2019;Lewis et al, 2020). However, several studies (Reimers and Gurevych, 2019;Li et al, 2020) have found that sentence embeddings from pre-trained language models perform poorly on semantic similarity tasks when the models are not fine-tuned on task-specific data. Meanwhile, Goldberg (2019) shows that BERT without fine-tuning performs surprisingly well on syntactic tasks.…”