BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models

He, Bin; Zhou, Di; Xiao, Jing; Jiang, Xin; Li, Qun; Yuan, Nicholas Jing; Xu, Tong

doi:10.18653/v1/2020.findings-emnlp.207

Cited by 73 publications

(64 citation statements)

References 25 publications

(23 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model is trained to distinguish the correct entity mention from randomly chosen ones. BERT-MK (He et al, 2020) integrates fact triples from knowledge graph. For each entity, it sample incoming and outcoming instances from the neighbors on the knowledge graph, and replaces head or tail entity to create negative instances.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, some efforts have been made to exploit injecting knowledge into pre-trained language models (Zhang et al, 2019;Lauscher et al, 2019;Levine et al, 2020;Peters et al, 2019;He et al, 2020;Xiong et al, 2020). Most previous works (as shown in Table 1) augment the standard language modeling objective with knowledge-driven objectives and update the entire model parameters.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Wang

Tang

Duan

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

134

View full text Add to dashboard Cite

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-ADAPTER, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-ADAPTER has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned texttriplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-ADAPTER captures versatile knowledge than RoBERTa. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Wang

Tang

Duan

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

134

View full text Add to dashboard Cite

show abstract

“…A brief comparison can be found in Table 1. CoLAKE is conceptually similar to K-BERT (Liu et al, 2020) and BERT-MK (He et al, 2019). CoLAKE differs from K-BERT in that, instead of injecting triplets during fine-tuning, CoLAKE jointly learns embeddings for entities and relations during pretraining LMs.…”

Section: Related Workmentioning

confidence: 99%

CoLAKE: Contextualized Language and Knowledge Embedding

Sun

Shao

Qiu

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

101

View full text Add to dashboard Cite

With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation. 1 * Work done during internship at Amazon Shanghai AI Lab.

show abstract

“…Lv et al (2020) propose a GNN-based inference model on conceptual network relationships and heterogeneous graphs of Wikipedia sentences. BERT-MK (He et al, 2019) integrates fact triples in the KG, while REALM (Guu et al, 2020) augments language model pre-training algorithms with a learned textual knowledge retriever. Unlike previous works, we incorporate external knowledge implicitly and explicitly.…”

Section: Related Workmentioning

confidence: 99%

“…In this section, we describe the method to extract knowledge facts from knowledge graph in details. Once the knowledge is determined, we can choose the appropriate integration mechanism for further knowledge injection, such as attention mechanism (Sun et al, 2018;Ma et al, 2019), pre-training tasks (He et al, 2019) and multi-task training . Given a question Q and a candidate answer O, we first identify the entity and its type in the text by entity linking.…”

Section: Knowledge Acquisitionmentioning

confidence: 99%

Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text

Li¹,

Hu²,

Chen³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Machine reading comprehension (MRC) has achieved significant progress on the open domain in recent years, mainly due to large-scale pre-trained language models. However, it performs much worse in specific domains such as the medical field due to the lack of extensive training data and professional structural knowledge neglect. As an effort, we first collect a large scale medical multi-choice question dataset (more than 21k instances) for the National Licensed Pharmacist Examination in China. It is a challenging medical examination with a passing rate of less than 14.2% in 2018. Then we propose a novel reading comprehension model KMQA, which can fully exploit the structural medical knowledge (i.e., medical knowledge graph) and the reference medical plain text (i.e., text snippets retrieved from reference books). The experimental results indicate that the KMQA outperforms existing competitive models with a large margin and passes the exam with 61.8% accuracy rate on the test set.

show abstract

BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models

Cited by 73 publications

References 25 publications

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

CoLAKE: Contextualized Language and Knowledge Embedding

Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text

Contact Info

Product

Resources

About