Relational World Knowledge Representation in Contextual Language Models: A Review

Safavi, Tara; Koutra, Danai

doi:10.18653/v1/2021.emnlp-main.81

Cited by 23 publications

(18 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5 that the CM method had the highest performance, but surprisingly, OM performed quite well. This highlights the ability of LMs to memorize the facts and act as soft KBs (Petroni et al, 2019;Safavi and Koutra, 2021). This trend is also consistent with general-domain .…”

Section: Discussionsupporting

confidence: 70%

MedDistant19: A Challenging Benchmark for Distantly Supervised Biomedical Relation Extraction

Amin¹,

Minervini²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relation Extraction in the biomedical domain is challenging due to the lack of labeled data and high annotation costs, needing domain experts. Distant supervision is commonly used as a way to tackle the scarcity of annotated data by automatically pairing knowledge graph relationships with raw texts. Distantly Supervised Biomedical Relation Extraction (Bio-DSRE) models can seemingly produce very accurate results in several benchmarks. However, given the challenging nature of the task, we set out to investigate the validity of such impressive results. We probed the datasets used by Amin et al. ( 2020) and Hogan et al. ( 2021) and found a significant overlap between training and evaluation relationships that, once resolved, reduced the accuracy of the models by up to 71%. Furthermore, we noticed several inconsistencies with the data construction process, such as creating negative samples and improper handling of redundant relationships. We mitigate these issues and present MEDDISTANT19, a new benchmark dataset obtained by aligning the MEDLINE abstracts with the widely used SNOMED Clinical Terms (SNOMED CT) knowledge base. We experimented with several state-of-the-art models achieving an AUC of 55.4% and 49.8% at sentence-and bag-level, showing that there is still plenty of room for improvement.

show abstract

Section: Discussionsupporting

confidence: 70%

MedDistant19: A Challenging Benchmark for Distantly Supervised Biomedical Relation Extraction

Amin¹,

Minervini²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Contemporary work has highlighted the promise of PLMs on high-level tasks requiring-among other things-access to proper relational knowledge between concepts (see Safavi and Koutra, 2021;Piantadosi and Hill, 2022). Findings from our experiments that target reasoning ability based on perhaps the most well-established of relationsthe ISA relation (Murphy, 2003)-suggest that PLMs' approximation of inference-making behavior based on simple relational knowledge is at best noisy, owing to clear failures in presence of distracting information.…”

Section: General Discussion and Conclusionmentioning

confidence: 99%

COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models

Misra¹,

Rayz²,

Ettinger³

2022

Preprint

View full text Add to dashboard Cite

A characteristic feature of human semantic memory is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (ANIMAL) to their subordinates (DOG)-i.e. demonstrate property inheritance. In this paper, we present COMPS, a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal that they can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the basis of nuanced knowledge representations. Furthermore, we find that PLMs can demonstrate behavior consistent with property inheritance to a great extent, but fail in the presence of distracting information, which decreases the performance of many models, sometimes even below chance. This lack of robustness in demonstrating simple reasoning raises important questions about PLMs' capacity to make correct inferences even when they appear to possess the prerequisite knowledge.

show abstract

“…LMs pretrained on a large corpus of web data have been shown to contain different kinds of knowledge implicitly in their parameters without the need for any human supervision. This includes: world knowledge (Petroni et al, 2019;Rogers et al, 2020), relational knowledge (Safavi and Koutra, 2021), commonsense knowledge (Da et al, 2021), linguistic knowledge (Peters et al, 2018;Goldberg, 2019;Tenney et al, 2019b), actionable knowledge * Equal Contribution † Equal Supervision 1 For an updated paper-list please check our website: https://bkhmsi.github.io/lms-as-kbs/.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, Wei et al (2021) evaluate knowledgeenhanced pretrained LMs by delineating the types of knowledge that can be integrated into existing LMs. Safavi and Koutra (2021) divide relevant work according to the level of supervision provided to the LM by a KB. Similarly, Colon-Hernandez et al (2021) cover the integration of structural knowledge into LMs but forgo implicit knowledge.…”

Section: Introductionmentioning

confidence: 99%

A Review on Language Models as Knowledge Bases

Badr¹,

Li²,

Çelikyılmaz³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem an LM should have to fully act as a KB, and review the recent literature with respect to those aspects. 1

show abstract

Relational World Knowledge Representation in Contextual Language Models: A Review

Cited by 23 publications

References 75 publications

MedDistant19: A Challenging Benchmark for Distantly Supervised Biomedical Relation Extraction

MedDistant19: A Challenging Benchmark for Distantly Supervised Biomedical Relation Extraction

COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models

A Review on Language Models as Knowledge Bases

Contact Info

Product

Resources

About