Multimodal Sentence Summarization via Multimodal Selective Encoding

Abstract

This paper studies the problem of generating a summary for a given sentence-image pair. Existing multimodal sequence-to-sequence approaches mainly focus on enhancing the decoder by visual signals, while ignoring that the image can improve the ability of the encoder to identify highlights of a news event or a document. Thus, we propose a multimodal selective gate network that considers reciprocal relationships between textual and multi-level visual features, including global image descriptor, activation grids, and object proposals, to select highlights of the event when encoding the source sentence. In addition, we introduce a modality regularization to encourage the summary to capture the highlights embedded in the image more accurately. To verify the generalization of our model, we adopt the multimodal selective gate to the text-based decoder and multimodal-based decoder. Experimental results on a public multimodal sentence summarization dataset demonstrate the advantage of our models over baselines. Further analysis suggests that our proposed multimodal selective gate network can effectively select important information in the input sentence.

Inter- and Intra-Modal Contrastive Hybrid Learning Framework for Multimodal Abstractive Summarization

“…MMAF and MMCF [ 55 ] are the modality-based attention mechanism for paying a different kind of attention to image patches and text units, which are filtered through selective visual information. Considering a selective gate network for reciprocal relationships between textual and multi-level visual features, SELECT [ 40 ] is the current SOTA baseline.…”

“…Considering visual information as a complement to textual features for generation [ 7 ], Zhu et al [ 39 ] propose a multimodal input and multimodal output dataset, as well as an attention model to generate a summary through a text-guided mechanism. The model Select [ 40 ] proposes a selective gate module for integrating reciprocal relationships, including a global image descriptor, activation grids and object proposals. Modeling the correlation among inputs is the core point of MAS.…”

CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization

“…[19] propose a multimodal attention model for How2 videos [25]. [13] design a gate to select event highlights from images and distinguishes highlights in encoding. [18] propose a local-global attention mechanism to let the video and text interact and select an image as output.…”

“…As for the text-only summarization, many works [26,4,3] aim to improve the word overlap metric, such as ROUGE. Similar to these works, most of the MS models also focus on improving the overlap toward the source text only [11,19,13], while the visual content is considered as the supplemental data handled independently. However, the visual content may convey some other information besides the textual content.…”

“…introduces a multimodal sentence summarization task that creates a condensed text summary from a long sentence and the corresponding image [19]. propose a multimodal attention model for How2 videos[25] [13]. design a gate to select event highlights from images and distinguishes highlights in encoding [18].…”