Exploiting Web Images for Video Highlight Detection With Triplet Deep Ranking

Kim, Hoseong; Mei, Tao; Byun, Hyeran; Yao, Ting

doi:10.1109/tmm.2018.2806224

Cited by 23 publications

(11 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gygli et al [6] used GIFs instead of manual labels as the ground truth for highlights. And Kim et al [5] proposed to detect keyframes with the help of Web images sharing the same topics with the corresponding videos. Audio Analysis for Video.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Detecting Highlighted Video Clips Through Emotion-Enhanced Audio-Visual Cues

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Recent years have witnessed the growing research interests in video highlight detection. Existing studies mainly focus on detecting highlights in user-generated videos with simple topics based on visual content. However, relying solely on visual features limits the ability of conventional methods to capture highlights for videos with more complicated semantics, like movies. Therefore, we propose to mine the emotional information in video sounds to enhance highlight detection. Specifically, we design a novel emotion-enhanced framework with multi-stage fusion to detect highlights for complex videos. Along this line, we first extract multi-grained features from the audio waves. Then, the tailored-designed intra-modal fusion is applied on audio features to obtain emotional representation. Furthermore, the cross-modal fusion is developed to generate comprehensive representation of clip by merging audio emotional representations and visual features. This representation can be leveraged for predicting highlight probability. Finally, extensive experiments on realworld datasets demonstrate the effectiveness of our method.

show abstract

Section: Related Workmentioning

confidence: 99%

“…These videos usually contain simple and clear content closely related to a certain topic, like swimming, climbing, and sky diving. As the importance of each clip is defined by its relevance to this topic in content [4,5], conventional highlight detection methods usually focus on mining * Tong Xu and Hui Xiong are corresponding authors.…”

Section: Introductionmentioning

confidence: 99%

Detecting Highlighted Video Clips Through Emotion-Enhanced Audio-Visual Cues

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

show abstract

“…Early proposed approaches employ various hand-crafted low-level visual features for highlight detection [16,[37][38][39]43]. Recently, various deep learning networks like DCNN [47] and LSTM [52] have been successfully applied for highlight extraction in general video summarization [17,46,47,52]. However, extracting highlights from a multi-camera broadcasting system has not been well explored.…”

Section: Related Workmentioning

confidence: 99%

Smart Director: An Event-Driven Directing System for Live Broadcasting

PanYingwei

ChenYue

BaoQian

et al. 2021

ACM Trans. Multimedia Comput. Commun. Appl.

Self Cite

View full text Add to dashboard Cite

Live video broadcasting normally requires a multitude of skills and expertise with domain knowledge to enable multi-camera productions. As the number of cameras keeps increasing, directing a live sports broadcast has now become more complicated and challenging than ever before. The broadcast directors need to be much more concentrated, responsive, and knowledgeable, during the production. To relieve the directors from their intensive efforts, we develop an innovative automated sports broadcast directing system, called Smart Director, which aims at mimicking the typical human-in-the-loop broadcasting process to automatically create near-professional broadcasting programs in real-time by using a set of advanced multi-view video analysis algorithms. Inspired by the so-called “three-event” construction of sports broadcast [ 14 ], we build our system with an event-driven pipeline consisting of three consecutive novel components: (1) the Multi-View Event Localization to detect events by modeling multi-view correlations, (2) the Multi-View Highlight Detection to rank camera views by the visual importance for view selection, and (3) the Auto-Broadcasting Scheduler to control the production of broadcasting videos. To our best knowledge, our system is the first end-to-end automated directing system for multi-camera sports broadcasting, completely driven by the semantic understanding of sports events. It is also the first system to solve the novel problem of multi-view joint event detection by cross-view relation modeling. We conduct both objective and subjective evaluations on a real-world multi-camera soccer dataset, which demonstrate the quality of our auto-generated videos is comparable to that of the human-directed videos. Thanks to its faster response, our system is able to capture more fast-passing and short-duration events which are usually missed by human directors.

show abstract

“…EEP metric learning (DML) has been used to compare the similarity of samples in a supervised or unsupervised manner, and has been applied to various fields such as product search [1,2], and video highlight detection [3]. A typical way of DML is to utilize the triplet loss that defines the triangular relationship between samples in terms of Euclidean distance [4,5].…”

Section: Introductionmentioning

confidence: 99%

Deep Metric Learning With Manifold Class Variability Analysis

Kim

Song

2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

In deep metric learning (DML) techniques, understanding both the local and global characteristics of embedding space is essential. However, conventional DML techniques have two limitations as follows: First, Euclidean distance-based metrics never imply global information such as class variability because they only depend on the physical distance of samples. Second, they assume that the embedding space is simply a vector space which cannot represent complex data features. Therefore, we propose a novel loss function which can fully utilize characteristics of embedding space by using discriminant analysis and nonlinear mapping. With theoretical analysis, the superior performance of the proposed method is verified for the fine-grained retrieval datasets such as Cars196, CUB200-2011, Stanford online products, and In-shop clothes.

show abstract

Exploiting Web Images for Video Highlight Detection With Triplet Deep Ranking

Cited by 23 publications

References 41 publications

Detecting Highlighted Video Clips Through Emotion-Enhanced Audio-Visual Cues

Detecting Highlighted Video Clips Through Emotion-Enhanced Audio-Visual Cues

Smart Director: An Event-Driven Directing System for Live Broadcasting

Deep Metric Learning With Manifold Class Variability Analysis

Contact Info

Product

Resources

About