Ranking Highlights in Personal Videos by Analyzing Edited Videos

Abstract

We present a fully automatic system for ranking domain-specific highlights in unconstrained personal videos by analyzing online edited videos. A novel latent linear ranking model is proposed to handle noisy training data harvested online. Specifically, given a targeted domain such as "surfing," our system mines the YouTube database to find pairs of raw and their corresponding edited videos. Leveraging the assumption that an edited video is more likely to contain highlights than the trimmed parts of the raw video, we obtain pair-wise ranking constraints to train our model. The learning task is challenging due to the amount of noise and variation in the mined data. Hence, a latent loss function is incorporated to mitigate the issues caused by the noise. We efficiently learn the latent model on a large number of videos (about 870 min in total) using a novel EM-like procedure. Our latent ranking model outperforms its classification counterpart and is fairly competitive compared with a fully supervised ranking system that requires labels from Amazon Mechanical Turk. We further show that a state-of-the-art audio feature mel-frequency cepstral coefficients is inferior to a state-of-the-art visual feature. By combining both audio-visual features, we obtain the best performance in dog activity, surfing, skating, and viral video domains. Finally, we show that impressive highlights can be detected without additional human supervision for seven domains (i.e., skating, surfing, skiing, gymnastics, parkour, dog activity, and viral video) in unconstrained personal videos.

Find Objects and Focus on Highlights: Mining Object Semantics for Video Highlight Detection via Graph Neural Networks

“…Performance Comparison From the results in Table 1-2, we have the following observations: YouTube Dataset: Table 1 summarizes the overall highlight detection results for different methods on YouTube dataset (Sun, Farhadi, and Seitz 2014) and shows that our proposed approach significantly outperforms the state-of-the-art methods in most domains. In particular, the accuracies of "parkour" and "skiing" achieve 0.83 and 0.69, which makes considerable improvements over others, as they do not utilize object semantic and relationship modeling techniques in video highlight detection.…”

“…YouTube dataset (Sun, Farhadi, and Seitz 2014): This dataset contains about 490 videos of six categories. The video is annotated as segment-level with three classes: 1-highlight; 0-normal; -1-non-highlight.…”

“…(2) LR (Sun, Farhadi, and Seitz 2014). It introduces latent variables to accommodate highlight detection and uses the EM-like self-paced model selection procedure to train the framework effectively.…”

Video Summarization: How to Use Deep-Learned Features Without a Large-Scale Dataset

“…The concept of learning to rank has been used for video summarization in a number of works [18,19,20]. The proposed approach, however, is entirely different from their methods in the following two aspects: First, we use the rank machine to enforce temporal evolution of CNN-learned frame features instead of scoring the importance of each frame trained based on the users preference.…”

Learning to Rank Based on Subsequences

“…This topic has been widely studied over the last years. Some applications include video analysis [35], person re-identification [38], zeroshot recognition [27], active learning [23], dimensionality reduction [9], 3D feature analysis [37], binary code learning [24], learning from privileged information [33], * The work was conducted at KU Leuven, ESAT-PSI. interestingness prediction [16] and action recognition using rank pooling [12]. In particular, we focus on image re-ranking.…”