Language Models as Knowledge Bases?

Petroni, Filippo; Rocktäschel, Tim; Lewis, Patrick; Bakhtin, Anton; Wu, Yuxiang; Miller, Alexander H.; Riedel, Sebastian

doi:10.18653/v1/d19-1250

Cited by 1,259 publications

(1,378 citation statements)

References 25 publications

Supporting

Mentioning

1,068

Contrasting

Unclassified

Order By: Relevance

“…Other examples of hybrid symbolic and sub-symbolic methods where a knowledge-base tool or graph-perspective enhances the neural (e.g., language [308]) model are in [309,310]. In reinforcement learning, very few examples of symbolic (graphical [311] or relational [75,312]) hybrid models exist, while in recommendation systems, for instance, explainable autoencoders are proposed [313].…”

Section: Hybrid Transparent and Black-box Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI

et al. 2020

View full text Add to dashboard Cite

In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.

show abstract

Section: Hybrid Transparent and Black-box Methodsmentioning

confidence: 99%

“…Neural-symbolic Systems [297,298,299,300] KB-enhanced Systems [24,169,301,308,309,310] Deep Formulation [264,302,303,304,305] Relational Reasoning [75,312,313,314] Case-base Reasoning [316,317,318] Fig. 11.a) ( Fig.…”

Section: Hybrid Transparent and Black-box Methodsmentioning

confidence: 99%

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Our work differs in that we consider arbitrary relations (as opposed to the object-property and object-affordance relations) and the fact that we automatically identify the most appropriate trigger sentences for each relation. The problem of extracting relational knowledge from the BERT language model was also studied very recently in Petroni et al (2019). In this work, a wide range of relations is considered, but their approach again depends on manually chosen trigger sentences.…”

Section: Related Workmentioning

confidence: 99%

Inducing Relational Knowledge from BERT

Bouraoui¹,

Camacho-Collados

Schockaert

2020

AAAI

125

115

View full text Add to dashboard Cite

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.

show abstract

“…Having defined our probing techniques, we now discuss how to generate the prompts for the recommendation and search probes, along with the templates we used. Based on the knowledge extracted from rating and review datasets, we create prompt sentences in a similar manner to how previous work extracted knowledge from other data sources [26,27].…”

Section: Templates and Prompt Generationmentioning

confidence: 99%

What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation

Penha

Hauff

2020

Fourteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Heavily pre-trained transformer models such as BERT have recently shown to be remarkably powerful at language modelling by achieving impressive results on numerous downstream tasks. It has also been shown that they are able to implicitly store factual knowledge in their parameters after pre-training. Understanding what the pre-training procedure of LMs actually learns is a crucial step for using and improving them for Conversational Recommender Systems (CRS). We first study how much off-the-shelf pre-trained BERT "knows" about recommendation items such as books, movies and music. In order to analyze the knowledge stored in BERT's parameters, we use different probes (i.e., tasks to examine a trained model regarding certain properties) that require different types of knowledge to solve, namely content-based and collaborative-based. Content-based knowledge is knowledge that requires the model to match the titles of items with their content information, such as textual descriptions and genres. In contrast, collaborative-based knowledge requires the model to match items with similar ones, according to community interactions such as ratings. Both are important types of knowledge for a CRS-a system that should ideally be able to explain recommendations, match a user's textual description of their information need to relevant items, and elucidate a user's interests. We resort to BERT's Masked Language Modelling (MLM) head to probe its knowledge about the genre of items, with cloze style prompts. In addition, we employ BERT's Next Sentence Prediction (NSP) head and representations' similarity (SIM) to compare relevant and non-relevant search and recommendation query-document inputs to explore whether BERT can, without any fine-tuning, rank relevant items first. The insights we gain from these probes help us understand BERT's limitations and strengths for conversational recommendation. Finally, we study how BERT performs in a conversational recommendation downstream task. To this end, we fine-tune BERT to act as a retrieval-based conversational recommender system. Overall, our analyses and experiments show that: (i) BERT has knowledge stored in its parameters about the content of books, movies and music; (ii) it has more content-based knowledge than collaborative-based knowledge; and (iii) fails on conversational recommendation when faced with adversarial data.

show abstract

Language Models as Knowledge Bases?

Cited by 1,259 publications

References 25 publications

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI

Inducing Relational Knowledge from BERT

What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation

Contact Info

Product

Resources

About