Predicting audio-visual salient events based on visual, audio and text modalities for movie summarization

Koutras, Petros; Zlatintsi, Athanasia; Iosif, Elias; Katsamanis, Athanasios; Maragos, Petros; Potamianos, Alexandros

doi:10.1109/icip.2015.7351630

Cited by 25 publications

(23 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Specifically, it is an improved synergistic approach to the problem of audio-visual salient event detection and movie summarization, also employing text saliency computation, initially presented in ICIP15 [14]. For the remainder of this paper, we refer to the two different frameworks as ICIP15 and TMM13.…”

Section: Multimodal Saliency-based Computational Frameworkmentioning

confidence: 99%

“…The 4 text features comprised the textual feature vector, where a second KNN model was built. The framework followed the same principles as explored in [12,95] and being further refined in ICIP15 [14]. Specifically, we considered frame-wise saliency as a two-class classification problem, while a confidence score was also determined for every classification result (i.e., each frame), in order to obtain results for various compression rates and thus produce summaries of various lengths.…”

Section: Machine Learning Evaluationmentioning

confidence: 99%

“…The evaluation verifies the appropriateness of the proposed methods and the quality of the produced summaries that consist of salient and semantically coherent events. Part of the experimental evaluation of this extended computational model was first presented in ICIP15 [14].…”

Section: Contributions and Overviewmentioning

confidence: 99%

“…For the spatio-temporal visual saliency estimation, we use an energy-based model [14,79], assumed to be more relevant to the cognition-inspired saliency methods [1,80]. For the extraction of visual features, our visual saliency model uses biologically plausible spatio-temporal 3D Gabor filters.…”

Section: Visual Analysismentioning

confidence: 99%

“…Four summaries, in total, were evaluated: two summaries based on the ICIP15 method [14] using different weights for the text modality, where w = 0.1 or 0.2, the best performing summary produced using the fusion methods (FUS) presented in TMM13 [12] (the summaries were chosen based on the best enjoyability results), and a fourth fast-forward like summary (FF), which was created by subsampling 2 seconds every 10 s of the original clip. The subjects participating in the evaluation first viewed the original half-hour clip, followed by the four summaries (ca.…”

Section: Subjective Qualitative Evaluation On the Hollywood Movie Summentioning

confidence: 99%

See 4 more Smart Citations

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

Zlatintsi

Koutras

Evangelopoulos

et al. 2017

J Image Video Proc.

View full text Add to dashboard Cite

Research related to computational modeling for machine-based understanding requires ground truth data for training, content analysis, and evaluation. In this paper, we present a multimodal video database, namely COGNIMUSE, annotated with sensory and semantic saliency, events, cross-media semantics, and emotion. The purpose of this database is manifold; it can be used for training and evaluation of event detection and summarization algorithms, for classification and recognition of audio-visual and cross-media events, as well as for emotion tracking. In order to enable comparisons with other computational models, we propose state-of-the-art algorithms, specifically a unified energy-based audio-visual framework and a method for text saliency computation, for the detection of perceptually salient events from videos. Additionally, a movie summarization system for the automatic production of summaries is presented. Two kinds of evaluation were performed, an objective based on the saliency annotation of the database and an extensive qualitative human evaluation of the automatically produced summaries, where we investigated what composes high-quality movie summaries, where both methods verified the appropriateness of the proposed methods. The annotation of the database and the code for the summarization system can be found at

show abstract

Section: Multimodal Saliency-based Computational Frameworkmentioning

confidence: 99%

Section: Machine Learning Evaluationmentioning

confidence: 99%

Section: Contributions and Overviewmentioning

confidence: 99%

Section: Visual Analysismentioning

confidence: 99%

Section: Subjective Qualitative Evaluation On the Hollywood Movie Summentioning

confidence: 99%

See 3 more Smart Citations

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

Zlatintsi

Koutras

Evangelopoulos

et al. 2017

J Image Video Proc.

View full text Add to dashboard Cite

show abstract

A topic modeling‐based bibliometric exploration of automatic summarization research

Chen,

Xie,

Tao

et al. 2024

WIREs Data Min & Knowl

View full text Add to dashboard Cite

The surge in text data has driven extensive research into developing diverse automatic summarization approaches to effectively handle vast textual information. There are several reviews on this topic, yet no large‐scale analysis based on quantitative approaches has been conducted. To provide a comprehensive overview of the field, this study conducted a bibliometric analysis of 3108 papers published from 2010 to 2022, focusing on automatic summarization research regarding topics and trends, top sources, countries/regions, institutions, researchers, and scientific collaborations. We have identified the following trends. First, the number of papers has experienced 65% growth, with the majority being published in computer science conferences. Second, Asian countries and institutions, notably China and India, actively engage in this field and demonstrate a strong inclination toward inter‐regional international collaboration, contributing to more than 24% and 20% of the output, respectively. Third, researchers show a high level of interest in multihead and attention mechanisms, graph‐based semantic analysis, and topic modeling and clustering techniques, with each topic having a prevalence of over 10%. Finally, scholars have been increasingly interested in self‐supervised and zero/few‐shot learning, multihead and attention mechanisms, and temporal analysis and event detection. This study is valuable when it comes to enhancing scholars' and practitioners' understanding of the current hotspots and future directions in automatic summarization.This article is categorized under: Algorithmic Development > Text Mining

show abstract

A comprehensive study of automatic video summarization techniques

Gupta

Sharma

2023

Artif Intell Rev

View full text Add to dashboard Cite

Predicting audio-visual salient events based on visual, audio and text modalities for movie summarization

Cited by 25 publications

References 30 publications

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

A topic modeling‐based bibliometric exploration of automatic summarization research

A comprehensive study of automatic video summarization techniques

Contact Info

Product

Resources

About