Compositional approaches for representing relations between words: A comparative study

Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as "eye is to seeing what ear is to hearing", sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations. 1

show abstract

“…Vector differences have been found to be the most robust encoding method in the context of word analogies(Hakami and Bollegala, 2017).…”

mentioning

confidence: 99%

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

Ushio¹,

Espinosa-Anke²,

Schockaert³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…The averaged token representation is taken as the term representation. A query or a candidate is estimated by the sum of the representations of each term pair, which is represented as the embedding vector differences (Hakami and Bollegala, 2017;Ushio et al, 2021). The candidate with the highest cosine similarity to the query is chosen as the answer.…”

Section: Baselines For Analogical Qamentioning

confidence: 99%

E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning

Chen¹,

Xu²,

Ziquan³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

The ability to recognize analogies is fundamental to human cognition. Existing benchmarks to test word analogy do not reveal the underneath process of analogical reasoning of neural models. Holding the belief that models capable of reasoning should be right for the right reasons, we propose a first-of-itskind Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (in Chinese) and 1,251 (in English) problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. More importantly, we design a free-text explanation scheme to explain whether an analogy should be drawn, and manually annotate them for each and every question and candidate answer. Empirical results suggest that this benchmark is very challenging for some state-of-the-art models for both explanation generation and analogical question answering tasks, which invites further research in this area. Project page of E-KAR can be found at https:// ekar-leaderboard.github.io.

show abstract

“…Bollegala et al (2015a) showed that PairDiff can be used as a proxy for learning better word embeddings and Vylomova et al (2016) conducted an extensive empirical comparison of PairDiff using a dataset containing 16 different relation types. Besides PairDiff, concatenation (Hakami and Bollegala, 2017;Yin and Schütze, 2016), circular correlation and convolution (Nickel et al, 2016) have been used in prior work for representing the relations between words. Because the relation embedding is composed using word embeddings instead of learning as a separate parameter, we refer to methods that are based on this approach as compositional relational embedding methods.…”

Section: Introductionmentioning

confidence: 99%

Why PairDiff works? -- A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection

Hakami,

Bollegala,

Kohei

2017

Preprint

Self Cite

View full text Add to dashboard Cite

Representing the semantic relations that exist between two given words (or entities) is an important first step in a wide-range of NLP applications such as analogical reasoning, knowledge base completion and relational information retrieval. A simple, yet surprisingly accurate method for representing a relation between two words is to compute the vector offset (PairDiff) between their corresponding word embeddings. Despite the empirical success, it remains unclear as to whether PairDiff is the best operator for obtaining a relational representation from word embeddings. We conduct a theoretical analysis of generalised bilinear operators that can be used to measure the 2 relational distance between two word-pairs. We show that, if the word embeddings are standardised and uncorrelated, such an operator will be independent of bilinear terms, and can be simplified to a linear form, where PairDiff is a special case. For numerous word embedding types, we empirically verify the uncorrelation assumption, demonstrating the general applicability of our theoretical result. Moreover, we experimentally discover PairDiff from the bilinear relation composition operator on several benchmark analogy datasets.

show abstract

Compositional approaches for representing relations between words: A comparative study

Cited by 11 publications

References 18 publications

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning

Why PairDiff works? -- A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection

Contact Info

Product

Resources

About