Video Highlights Detection and Summarization with Lag-Calibration based on Concept-Emotion Mapping of Crowdsourced Time-Sync Comments

Abstract

With the prevalence of video sharing, there are increasing demands for automatic video digestion such as highlight detection. Recently, platforms with crowdsourced time-sync video comments have emerged worldwide, providing a good opportunity for highlight detection. However, this task is non-trivial: (1) time-sync comments often lag behind their corresponding shot; (2) time-sync comments are semantically sparse and noisy; (3) to determine which shots are highlights is highly subjective. The present paper aims to tackle these challenges by proposing a framework that (1) uses concept-mapped lexical-chains for lagcalibration; (2) models video highlights based on comment intensity and combination of emotion and concept concentration of each shot; (3) summarize each detected highlight using improved SumBasic with emotion and concept mapping. Experiments on large real-world datasets show that our highlight detection method and summarization method both outperform other benchmarks with considerable margins.

Knowing Where and What to Write in Automated Live Video Comments: A Unified Multi-Task Approach

“…Multimedia highlight detection is a wide ranging topic that extends beyond danmu commenting. Much work has been done in detecting highlights for sports videos [1,14,24], game videos [16] and, most commonly, user-generated web videos [2,9,15,18,23,28]. Extracting the best image thumbnail [17,26] from videos is for instance of the highest relevance for video sharing platforms.…”

“…As danmu comments often lag behind their corresponding video scene [15], we smooth the comment distribution by applying moving average over a 5sec window to alleviate the lag effect and better align the comment distribution with the video content.…”

Promoting Mental Well-Being for Audiences in a Live-Streaming Game by Highlight-Based Bullet Comments

“…We utilize an open data set collected from Bilibili (Chinese video-sharing website with bullet comments) introduced by Qing et al [9] as our highlight bullet comments. It contained emotional words often used in bullet comments.…”

Interactive Variance Attention based Online Spoiler Detection for Time-Sync Comments

“…Yang et al [34] systematically collate the features of TSCs and introduces a graphbased algorithm to eliminate the impact of noise prominently. Then further usages of TSCs data are proposed like extracting highlight shots [24,30], labeling important segments [19], detecting events [16], generating temporal descriptions of videos [31], and video recommendation [5,23,32]. Besides, some other researchers turn to build and improve TSC dataset.…”