MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention

Abstract

This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -text, audio and video -in a multimodal video. Prior work on multimodal abstractive text summarization only utilized information from the text and video modalities. We examine the usefulness and challenges of deriving information from the audio modality and present a sequence-to-sequence trimodal hierarchical attention-based model that overcomes these challenges by letting the model pay more attention to the text modality. MAST outperforms the current state of the art model (video-text) by 2.51 points in terms of Content F1 score and 1.00 points in terms of Rouge-L score on the How2 dataset for multimodal language understanding.* indicates equal contribution Aman Khullar is presently at Gram Vaani 61Original text: let's talk now about how to bait a tip up hook with a maggot. typically, you're going to be using this for pan fish. not a real well known or common technique but on a given day it could be the difference between not catching fish and catching fish. all you do, you take your maggot, you can use meal worms, as well, which are much bigger, which are probably more well suited for this because this is a rather large hook. you would just, again, put that hook right through the maggot. with a big hook like this, i would probably put ten of these on it, just line the whole thing. this is going to be more of a technique for pan fish, such as, perch and sunfish, some of your smaller fish but if you had maggots, like this , or a meal worm, or two, on a hook like this, this would be a fantastic setup for trout, as well.Text only: ice fishing is used for ice fishing. learn about ice fishing bait with tips from an experienced fisherman artist in this free fishing video.Video-Text: learn about the ice fishing bait in this ice fishing lesson from an experienced fisherman.MAST: maggots are good for catching perch. learn more about ice fishing bait in this ice fishing lesson from an experienced fisherman.

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

“…For each video clip, following previous works (Sanabria et al, 2018;Palaskar et al, 2019;Khullar and Arora, 2020), a 2048-dimensional feature representation is extracted for every 16 non-overlapping frames using a 3D ResNeXt-101 model (Hara et al, 2018), which is pre-trained on the Kinetics dataset (Kay et al, 2017). Therefore, each data sample will have a sequence of 2048-vision feature vectors of length .…”

MCR: Multilayer cross‐fusion with reconstructor for multimodal abstractive summarisation

“…[1] first proposed to utilise the visual and textual information of video clips into the summary generation process. Khullar and Arora [26] incorporated audio to generate a summary of video content with visual and textual modalities. Liu et al.…”

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

“…For each video clip, following previous works (Sanabria et al, 2018;Palaskar et al, 2019;Khullar and Arora, 2020), a 2048-dimensional feature representation is extracted for every 16 non-overlapping frames using a 3D ResNeXt-101 model (Hara et al, 2018), which is pre-trained on the Kinetics dataset (Kay et al, 2017). Therefore, each data sample will have a sequence of 2048-𝑑 vision feature vectors of length 𝑀.…”