Learning beyond Datasets: Knowledge Graph Augmented Neural Networks for Natural Language Processing

Abstract

Machine Learning has been the quintessential solution for many AI problems, but learning models are heavily dependent on specific training data. Some learning models can be incorporated with prior knowledge using a Bayesian setup, but these learning models do not have the ability to access any organized world knowledge on demand. In this work, we propose to enhance learning models with world knowledge in the form of Knowledge Graph (KG) fact triples for Natural Language Processing (NLP) tasks. Our aim is to develop a deep learning model that can extract relevant prior support facts from knowledge graphs depending on the task using attention mechanism. We introduce a convolutionbased model for learning representations of knowledge graph entity and relation clusters in order to reduce the attention space. We show that the proposed method is highly scalable to the amount of prior information that has to be processed and can be applied to any generic NLP task. Using this method we show significant improvement in performance for text classification with 20Newsgroups (News20) & DBPedia datasets, and natural language inference with Stanford Natural Language Inference (SNLI) dataset. We also demonstrate that a deep learning model can be trained with substantially less amount of labeled training data, when it has access to organized world knowledge in the form of a knowledge base.

MG-BERT: Multi-Graph Augmented BERT for Masked Language Modeling

“…Yao et al (2019) employ KG-BERT in triple classification, link prediction, and relation prediction tasks. Furthermore, knowledge graphs are used in NLP tasks such as text classification (K M et al, 2018;Ostendorff et al, 2019;, named entity recognition (Dekhili et al, 2019), and language modeling (Ahn et al, 2016;Logan et al, 2019). ERNIE (Zhang et al, 2019b) is an enhanced language representation model incorporating knowledge graphs.…”

Probing Pre-trained Language Models for Semantic Attributes and their Values

“…Knowledge bases (KBs), such as Wikidata (Vrandečić and Krötzsch, 2014), constitute a valuable resource for collecting attributes and their values. In general, KBs have been shown to help improve multiple NLP application as they contain structured information (Annervaz et al, 2018;Nakashole and Mitchell, 2015;Rahman and Ng, 2011;Ratinov and Roth, 2009). As matter of fact, it is fairly simple to answer factoid questions such as "How old is Joe Biden?"…”

Does Commonsense help in detecting Sarcasm?

“…However, none of the previous works have explored the idea of incorporating commonsense knowledge in sarcasm detection. Common sense has been used in several natural-language based tasks like controllable story generation (Zhang et al, 2020;Brahman and Chaturvedi, 2020), sentence classification (Chen et al, 2019), question answering (Dzendzik et al, 2020), natural language inference (K M et al, 2018;Wang et al, 2019) and other related tasks but not for sarcasm detection. We hypothesize that commonsense knowledge, capturing general beliefs and world knowledge, can prove instrumental in understanding sarcasm.…”