MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention

Abstract

This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -text, audio and video -in a multimodal video. Prior work on multimodal abstractive text summarization only utilized information from the text and video modalities. We examine the usefulness and challenges of deriving information from the audio modality and present a sequence-to-sequence trimodal hierarchical attention-based model that overcomes these challenges by letting the model pay more attention to the text modality. MAST outperforms the current state of the art model (video-text) by 2.51 points in terms of Content F1 score and 1.00 points in terms of Rouge-L score on the How2 dataset for multimodal language understanding.

HGTMFS: A Hypergraph Transformer Framework for Multimodal Summarization

“…Multimodal summarization focuses on distilling the most significant aspects of input data that span multiple modalities. Recent years have seen extensive exploration into generating text summaries from multimodal data, which may include text, visual, and auditory information [19][20][21]. A significant body of work has concentrated on incorporating visual information to enhance the quality of text summa-rization [3][4][5]22,23].…”

MCR: Multilayer cross‐fusion with reconstructor for multimodal abstractive summarisation

“…Based on the How2, Palaskar et al [1] first proposed to utilise the visual and textual information of video clips into the summary generation process. Khullar and Arora [26] incorporated audio to generate a summary of video content with visual and textual modalities. Liu et al [3] conducted multistage fusion to interact multi-source modalities together and applied the forget gate module to resist the noise flows from multimodal semantics.…”

CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization

“…dle the text input.Multimodal summarization aims to condense information from multimodal inputs, such as text, vision, and audio[12]. Recently, Ms has been extensively studied ([10,2,8,31]). A large number of the works focus on fusing visual information to improve the quality of text summaries [11].…”