Reading the Videos: Temporal Labeling for Crowdsourced Time-Sync Videos Based on Semantic Embedding

Abstract

Recent years have witnessed the boom of online sharing media contents, which raise significant challenges in effective management and retrieval. Though a large amount of efforts have been made, precise retrieval on video shots with certain topics has been largely ignored. At the same time, due to the popularity of novel time-sync comments, or so-called "bullet-screen comments", video semantics could be now combined with timestamps to support further research on temporal video labeling. In this paper, we propose a novel video understanding framework to assign temporal labels on highlighted video shots. To be specific, due to the informal expression of bullet-screen comments, we first propose a temporal deep structured semantic model (T-DSSM) to represent comments into semantic vectors by taking advantage of their temporal correlation. Then, video highlights are recognized and labeled via semantic vectors in a supervised way. Extensive experiments on a real-world dataset prove that our framework could effectively label video highlights with a significant margin compared with baselines, which clearly validates the potential of our framework on video understanding, as well as bullet-screen comments interpretation.

Topic-based Video Analysis

“…Dieng et al [30] have presented a topic-based recurrent neural network (RNN) for sentiment analysis. Lv et al [98] have used LDA and deep learning to describe videos using language. Recently, Dong et al [32] have used LDA-based topic discovery and learning to produce interpretable deep learning for video description.…”

“…The bag-of-words is constructed using low-level features such as pixels [43] or motion tracks [152]. The problem of representing similar concepts using similar bag-of-words is solved using the contextual relevance representation [59,98]. The method uses language embedded with visual words for finding the similar concepts and applied in video analysis.…”

A Context-Enriched Neural Network Method for Recognizing Lexical Entailment

“…Intuitively, each embedding matrix can compress the vocabulary into a low-dimensional space (Li et al 2015;Lv et al 2016). However, because the multiple embedding spaces are constructed based on different contexts in the text corpus, the embedding spaces are different from each other.…”

Bridging Video Content and Comments: Synchronized Video Description with Temporal Summarization of Crowdsourced Time-Sync Comments

“…Efforts have also been devoted to associate comments with video content along the timeline. In (Lv et al 2016), time-sync comments are first represented with semantic vectors, then a video splitting framework is designed to extract and label meaningful segments based on mapping the semantic vectors to pre-defined labels in a supervised way. However, this model relies on a large amount of human-labeled video segments and predefined emotional tags to train, which limits its applicability to more general scenarios.…”

“…Recently, methods have been proposed to generate temporal tags or labels based on crowdsourced time-sync video comments Lv et al 2016), which mainly focus on extracting keywords such as topics or semantic labels. On the other hand, keywords sometimes are not sufficient to describe a scene, especially when the scene includes a number of characters or depicts a complicated situation.…”