Multi-modal Multi-relational Feature Aggregation Network for Medical Knowledge Representation Learning

Abstract

Representation learning of medical Knowledge Graph (KG) is an important task and forms the fundamental process for intelligent medical applications such as disease diagnosis and healthcare question answering. Therefore, many embedding models have been proposed to learn vector presentations for entities and relations but they ignore three important properties of medical KG: multimodal, unbalanced and heterogeneous. Entities in the medical KG can carry unstructured multi-modal content, such as image and text. At the same time, the knowledge graph consists of multiple types of entities and relations, and each entity has various number of neighbors. In this paper, we propose a Multi-modal Multi-Relational Feature Aggregation Network (MMRFAN) for medical knowledge representation learning. To deal with the multi-modal content of the entity, we propose an adversarial feature learning model to map the textual and image information of the entity into the same vector space and learn the multi-modal common representation. To better capture the complex structure and rich semantics, we design a sampling mechanism and aggregate the neighbors with intra and interrelation attention. We evaluate our model on three knowledge graphs, including FB15k-237, IMDb and Symptoms-in-Chinese with link prediction and node classification tasks. Experimental results show that our approach outperforms state-of-the-art method. CCS CONCEPTS • Computing methodologies → Knowledge representation and reasoning.

Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

“…MKHAN [51] proposes a hierarchical atention network on multi-modal medical knowledge graphs to deal with the explainable medical question answering task. MMRFAN [50] proposes an adversarial feature learning model to map the textual and image information of the entity into the same vector space and capture the multimodal information with a multi-relational feature aggregation network. Sun et al [36] propose a multi-modal graph atention network to optimize the recommendation system with multi-modal knowledge graph.…”

“…• MMRFAN [50]: MMRFAN is a GNN-based model which learns multi-modal information for entities with an adversarial feature learning module, and then uses relational graph convolution operation for multimodel knowledge graph embedding.…”

“…MKHAN [51] proposes a hierarchical atention network to encode the multi-modal information in a medical knowledge graph for explainable medical question answering. MMRFAN [50] proposes a GNN-based model for multi-modal medical knowledge representation learning.…”

“…However, the above multi-modal knowledge graph embedding methods still cannot deal with the following challenges: (1) Entities with multi-modal information have intra-modal and inter-modal dynamics, but traditional methods [44,51] with simple concatenation or atention cannot deal with this property. Although MM-RFAN [50] uses adversarial feature learning, it cannot be trained end-to-end. (2) As shown in Figure 1, the multi-modal information of entities is unbalanced, e.g., Musical Film has no numerical information because it is a movie genre descriptor.…”

“…(3) To capture the graph structure information, we propose an information aggregation block. Compared with traditional graph neural network [50], we use relation-speciic atention and entity-relation fusion operation to improve representation learning for multi-modal knowledge graph. At last, in the prediction block we use ConvE [13] as a decoder to calculate the credibility of triplets based on the multi-modal knowledge graph embeddings.…”