Text-Image-Video Summary Generation Using Joint Integer Linear Programming

Abstract

Automatically generating a summary for asynchronous data can help users to keep up with the rapid growth of multi-modal information on the Internet. However, the current multi-modal systems usually generate summaries composed of text and images. In this paper, we propose a novel research problem of text-image-video summary generation (TIVS). We first develop a multi-modal dataset containing text documents, images and videos. We then propose a novel joint integer linear programming multi-modal summarization (JILP-MMS) framework. We report the performance of our model on the developed dataset.

A Survey on Multi-modal Summarization

“…Learning process (LP): A lot of work has been done in both supervised learning [13,57,62,133,134] and unsupervised learning [25,26,[44][45][46]58]. It can be observed that a large fraction of supervised techniques adopt deep neural networks to tackle the problem [12,57,62,133], whereas in unsupervised techniques a large diversity of techniques have been adopted including deep neural networks [13], integer linear programming [44], differential evolution [45,46], submodular optimization [58] etc.…”

“…Since a major focus of this survey is on MMS tasks with text as the central modality, the number of text documents in input can also be one way of categorizing the related works. Depending upon whether the textual input is single-document [13,57,133] or multi-document [44][45][46]58], the summarization strategies might differ.…”

“…Kind of text summary (KTS): The most widely discussed distinction for text summarization works is the distinction of extractive vs abstractive. Similarly, depending on the nature of an output text summary, we can also classify the works in MMS tasks (containing text in the output) into extractive MMS [13,[44][45][46]58] and abstractive MMS [12,57,133,134] 6 .…”

“…Base modality (BS): Based on central-modality (defined in Section 2), existing works can also be distinguished depending on the base modality around which the final output as well as the model are formulated. A large potion of the prior work adopts either a text-centric approach [12,44,46,57,58,62,133] or a video-centric 7 approach [25,26,95,115].…”

“…This alarmingly increasing amount of content on the Internet makes it difficult for the users to receive useful information from the torrent of sources, necessitating research on the task of multi-modal summarization (MMS). Various studies have shown that including multi-modal data as input can indeed help improve the summary quality [44,58]. Zhu et al [133] claimed that on an average having a pictorial summary can improve the user satisfaction by 12.4% over a plain text summary.…”