Recognizing and Presenting the Storytelling Video Structure With Deep Multimodal Networks

Abstract

Abstract-This paper presents a novel approach for temporal and semantic segmentation of edited videos into meaningful segments, from the point of view of the storytelling structure. The objective is to decompose a long video into more manageable sequences, which can in turn be used to retrieve the most significant parts of it given a textual query and to provide an effective summarization. Previous video decomposition methods mainly employed perceptual cues, tackling the problem either as a story change detection, or as a similarity grouping task, and the lack of semantics limited their ability to identify story boundaries. Our proposal connects together perceptual, audio and semantic cues in a specialized deep network architecture designed with a combination of CNNs which generate an appropriate embedding, and clusters shots into connected sequences of semantic scenes, i.e. stories. A retrieval presentation strategy is also proposed, by selecting the semantically and aesthetically "most valuable" thumbnails to present, considering the query in order to improve the storytelling presentation. Finally, the subjective nature of the task is considered, by conducting experiments with different annotators and by proposing an algorithm to maximize the agreement between automatic results and human annotators.

A Video Library System Using Scene Detection and Automatic Tagging

“…In particular, our clustering algorithm relies on the minimization of variances inside each scene. For further details, the reader is encouraged to read the paper in which the technique was proposed [5].…”

“…Using a scene detection algorithm that we have recently proposed in literature [5], and thanks to the application of Speech-to-Text techniques, it has been possible to automatically annotate a set of 500 educational broadcast videos taken from the large Rai Scuola archive 2 . Also, we developed a browsing and retrieval interface on top of a commercial ECMS, namely eXo Platform, from which the results of the automatic annotation can be browsed and manually refined.…”

Recuperação de Informação Multimídia em Big Data Utilizando OpenCV Python

“…Atualmente, as pessoas podem ter acesso a conteúdos usando diferentes tipos de dispositivos e meios (notebooks, celulares, Personal Digital Assistants, WiFi, 3G e 4G, entre outros). Toda essa evolução criou ambientes heterogêneos [Bouyakoub e Belkhir, 2008;Baraldi et al, 2017;Pouyanfar et al, 2018], surgindo com isso desafios no tratamento dos dados, já que, geralmente, quando um dispositivo acessa um conteúdo multimídia para o qual não foi projetado, a experiência do usuário é insatisfatória.…”

Annote: A Serious Game for Medical Students to Approach Lesion Skin Images of a Digital Library

“…The Gamification process consists in the application of game-design elements and principles in non-game contexts [16]: it uses the game mechanics to improve skills and knowledge of a subject, also enhancing its engagement and excitement while performing a task that usually does not provides them. Referring to the Csíkszentmihályi [12,19] and Chen [8] studies the sense of fun is strictly connected with the Flow theory characterized by the constant steady and balance between the challenge offered to gamers and the skills developed while facing them: in [5,11,13] are studies about video semantic recognition while the evaluation of affective states and moods are in [3].…”