IsoScore: Measuring the Uniformity of Embedding Space Utilization

Rudman, William J.; Gillman, Nate; Rayne, Taylor; Eickhoff, Carsten

doi:10.18653/v1/2022.findings-acl.262

Cited by 5 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, methods that are based on the Principal Component Analysis (PCA) are the most appropriate to find and study the most elongated directions of the space. We present the two most robust PCA-based methods to quantify the isotropy, mainly the explained variance ratio and the IsoScore as highlighted in [21].…”

Section: Methodsmentioning

confidence: 99%

“…The IsoScore [21] of an embedding space can be interpreted as the fraction of dimensions uniformly used by the embedding space. This score is derived from an isotropy defect that is calculated by computing the distance between the identity matrix and the normalized covariance matrix of the PCA-reoriented data.…”

Section: Isoscorementioning

confidence: 99%

“…Table 1 presents a comparison between the Explained Variance and the IsoScore, as presented in [21].…”

Section: Isoscorementioning

confidence: 99%

See 2 more Smart Citations

Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

Attieh,

Woubie Zewoudie,

Vlassov

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Recent studies show that the spatial distribution of the sentence representations generated from pre-trained language models is highly anisotropic, meaning that the representations are not uniformly distributed among the directions of the embedding space. Thus, the expressiveness of the embedding space is limited, as the embeddings are less distinguishable and less diverse. This results in a degradation in the performance of the models on the downstream task. Most methods that define the state-of-the-art in this area proceed by improving the isotropy of the sentence embeddings by refining the corresponding contextual word representations, then deriving the sentence embeddings from these refined representations. In this thesis, we propose to improve the quality and distribution of the sentence embeddings extracted from the [CLS] token of the pre-trained language models by improving the isotropy of the embeddings. We add one feed-forward layer, referred to as the Isotropy Layer, between the model and the downstream task layers. We train this layer using a novel joint loss function that optimizes an isotropy quality measure and the downstream task loss. This joint loss pushes the embeddings outputted by the Isotropy Layer to be more isotropic, and it also retains the semantics needed to perform the downstream task. The proposed approach results in transformed embeddings with better isotropy, that generalize better on the downstream task. Furthermore, the approach requires training one feed-forward layer, instead of retraining the whole network. We quantify and evaluate the isotropy through multiple metrics, mainly the Explained Variance and the IsoScore. Experimental results on 3 GLUE datasets with classification as the downstream task show that our proposed method is on par with the state-of-the-art, as it achieves performance gains of around 2-3% on the downstream tasks compared to the baseline. We also present a small case study on one language abuse detection dataset, then interpret some of the findings in light of the results.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Isoscorementioning

confidence: 99%

See 1 more Smart Citation

Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

Attieh,

Woubie Zewoudie,

Vlassov

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…For similarity metric, we experiment with cosine similarity, as well as a more recent approach -IsoScore (Rudman et al, 2022). In our experiments, we find that cosine similarity performs better overall, so the results reported in the paper are using cosine similarity.…”

Section: Stratified Example Retrieval For Supervisionmentioning

confidence: 96%

Exploring Distributional Shifts in Large Language Models for Code Analysis

Arakelyan,

Das,

Mao

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We systematically study how three large language models with code capabilities -CodeT5, Codex, and ChatGPT -generalize to out-ofdomain data. We consider two fundamental applications -code summarization, and code generation. We split data into domains following its natural boundaries -by an organization, by a project, and by a module within the software project. We establish that samples from each new domain present all the models with a significant challenge of distribution shift. We study how established methods adapt models to better generalize to new domains. Our experiments show that while multitask learning alone is a reasonable baseline, combining it with few-shot finetuning on examples retrieved from training data can achieve very strong performance. Moreover, this solution can outperform direct finetuning for very low-data scenarios. Finally, we consider variations of this approach to create a more broadly applicable method to adapt to multiple domains at once. We find that for code generation, a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain 1 .

show abstract

“…Isotropy is often estimated by different ways: singular value decomposition (SVD) (Biś et al, 2021;Gao et al, 2019;Liang et al, 2021;Wang et al, 2020a), intrinsic dimension (Cai et al, 2021), partition function (Arora et al, 2016;Mu and Viswanath, 2018), average cosine similarity (Ethayarajh, 2019). We chose the two firsts, along with IsoScore (Rudman et al, 2022) that complement themselves. We did not measure isotropy on models using embedding pooling, as it would be untractable considering the very large number of possible embeddings, and that the low frequency of the majority of them would result in unreliable results.…”

Section: Learned Embedding Spacesmentioning

confidence: 99%

Byte Pair Encoding for Symbolic Music

Fradet,

Gutowski,

Chhel

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

When used with deep learning, the symbolic music modality is often coupled with language model architectures. To do so, the music needs to be tokenized, i.e. converted into a sequence of discrete tokens. This can be achieved by different approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations rely on small vocabularies of tokens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, we show that Byte Pair Encoding, a compression technique widely used for natural language, significantly decreases the sequence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expressive tokens, resulting in both better results and faster inference in generation and classification tasks. The source code is shared on Github 1 , along with a companion website 2 . Finally, BPE is directly implemented in MidiTok 3 , allowing the reader to easily benefit from this method.

show abstract

IsoScore: Measuring the Uniformity of Embedding Space Utilization

Cited by 5 publications

References 7 publications

Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

Exploring Distributional Shifts in Large Language Models for Code Analysis

Byte Pair Encoding for Symbolic Music

Contact Info

Product

Resources

About