Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension

Abstract

Machine reading comprehension (MRC) is a crucial and challenging task in NLP. Recently, pre-trained language models (LMs), especially BERT, have achieved remarkable success, presenting new state-of-the-art results in MRC. In this work, we investigate the potential of leveraging external knowledge bases (KBs) to further improve BERT for MRC. We introduce KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context-and knowledgeaware predictions. We believe this would combine the merits of both deep LMs and curated KBs towards better MRC. Experimental results indicate that KT-NET offers significant and consistent improvements over BERT, outperforming competitive baselines on ReCoRD and SQuAD1.1 benchmarks. Notably, it ranks the 1st place on the ReCoRD leaderboard, and is also the best single model on the SQuAD1.1 leaderboard at the time of submission (March 4th, 2019). 1 * This work was done while the first author was an intern at Baidu Inc.

Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report

“…Table 4 shows EM (%) and F 1 (%) of human performance, the PSH-SJTU system as well as baselines on the development and test sets of task 2. Compared with the best baseline, KT-NET (Yang et al, 2019a), PSH-SJTU achieves significantly better scores. On the hidden test set, they improve EM by 10.08%, and F 1 by 8.98%.…”

“…KT-NET (Yang et al, 2019a) employs an attention mechanism to adaptively select desired knowledge from knowledge bases, and then fuses selected knowledge with BERT to enable contextand knowledge-aware predictions for machine reading comprehension. (Seo et al, 2016) and self-attention, both of which are widely used in MC models.…”

Towards Knowledge Enhanced Language Model for Machine Reading Comprehension

“…The first one, mutual attention, is aimed at fusing the question representations into the passage so as to obtain the question-aware passage representations; the second one, self-attention, is aimed at fusing the question-aware passage representations into themselves so as to obtain the final passage representations. Yang et al [35] proposed KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context-and knowledge-aware predictions. However, most previous methods retrieve the relevant knowledge base before encoding it into the MRC, which can only refer to KB locally.…”

Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering

“…This has led many to leverage knowledge graphs (KGs) (Mihaylov and Frank, 2018;Lin et al, 2019;Yang et al, 2019). KGs represent relational knowledge between entities with multi-relational edges for models to acquire.…”