ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene Graphs

Abstract

We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.

Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge

“…For multi-modal fusion modules, existing methods can be classified into two categories (i.e., single-stream and dual-stream). In specific, for the single-stream fusion, the models [8,27,28,43] use a single Transformer for early and unconstrained fusion between modalities; for the dual-stream fusion, the models [35,47,55] adopt the co-attention mechanism to interact different modalities. For pretext tasks, inspired by uni-modal pre-training schemes such as MLM [10,33] and causal language modeling [6], existing studies explore a variety of pre-training tasks, including MLM [27,35,47], MIM [8,35], ITM [27,58], image-text contrastive [26] and prefix language modeling [51].…”

“…According to the knowledge injection schemes, existing studies can be classified into four categories: embeddings combination [41,59], data structure compatibility [16,32,45], knowledge supervision [46,49], and neural-symbolic methods [2]. For VLP, knowledge can be acquired from both the image and text modalities, and there are several works [9,28,55] studying to integrate knowledge into their methods. ERNIE-ViL [55] built detailed semantic alignments between vision and language based on the scene graph parsed from the text.…”

“…Although these methods have motivated the learning of imagetext correspondences through well-designed model architectures and pretext tasks, most of them disregard the complementary information (i.e., knowledge) shared by different modalities and still lack the explicit knowledge modeling for Med-VLP. Even in the general domain, there are only a few VLP studies [7,9,28,55] on incorporating external knowledge into the pre-training process. For instance, ERNIE-ViL [55] constructed a scene graph from the input text to build the semantic connections between vision and language and emphasized the importance of keywords (e.g., objects, attributes, and relationships between objects) through the designs of pretext tasks.…”

“…Even in the general domain, there are only a few VLP studies [7,9,28,55] on incorporating external knowledge into the pre-training process. For instance, ERNIE-ViL [55] constructed a scene graph from the input text to build the semantic connections between vision and language and emphasized the importance of keywords (e.g., objects, attributes, and relationships between objects) through the designs of pretext tasks. ROSITA [9] used a unified scene graph shared by the input image and text to enhance the semantic alignments between vision and language.…”

MMASleepNet: A multimodal attention network based on electrophysiological signals for automatic sleep staging

“…Recurrent Neural Networks (RNNS) were proposed to capture the temporal correlation of electrophysiological signals (Michielli et al, 2019 ). Attention mechanism and attention-based feature fusion have been widely used in multimodal representation learning (Huang et al, 2019 , 2020 ; Lu et al, 2019 ; Wei et al, 2020 ; Zhang et al, 2020a , b , c ; Desai and Johnson, 2021 ; Yu et al, 2021 ; Ma et al, 2022 ). The existing studies based on attention mechanisms usually used single-modal data such as EEG or EOG, which only focused on the inter-relationship among single modality features rather than cross-modal features (Eldele et al, 2021 ).…”