Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval

Abstract

With the explosive growth of web videos in recent years, large-scale Content-Based Video Retrieval (CBVR) becomes increasingly essential in video filtering, recommendation, and copyright protection. Segment-level CBVR (S-CBVR) locates the start and end time of similar segments in finer granularity, which is beneficial for user browsing efficiency and infringement detection especially in long video scenarios. The challenge of S-CBVR task is how to achieve high temporal alignment accuracy with efficient computation and low storage consumption. In this paper, we propose a Segment Similarity and Alignment Network (SSAN) in dealing with the challenge which is firstly trained end-to-end in S-CBVR. SSAN is based on two newly proposed modules in video retrieval: (1) An efficient Self-supervised Keyframe Extraction (SKE) module to reduce redundant frame features, (2) A robust Similarity Pattern Detection (SPD) module for temporal alignment. In comparison with uniform frame extraction, SKE not only saves feature storage and search time, but also introduces comparable accuracy and limited extra computation time. In terms of temporal alignment, SPD localizes similar segments with higher accuracy and efficiency than existing deep learning methods. Furthermore, we jointly train SSAN with SKE and SPD and achieve an end-to-end improvement. Meanwhile, the two key modules SKE and SPD can also be effectively inserted into other video retrieval pipelines and gain considerable performance improvements. Experimental results on public datasets show that SSAN can obtain higher alignment accuracy while saving storage and online query computational cost compared to existing methods.

A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detection

“…Inspired by temporal matching kernel [27], LAMV [7] transforms the kernel into a differentiable layer to find temporal alignments. SPD [8] formulates temporal alignment as an object detection task on the frame-to-frame similarity matrix, achieving a state-of-theart segment-level copy detection performance.…”

“…Previous segment-level evaluation metrics are introduced with MUSCLE-VCD [15] and VCDB datasets [11]. Most of recent research works [7][8][9] adopt segment precision and recall defined in VCDB as follows:…”

“…In most cases, video-level copy detection results alone are not sufficient as the detected videos are usually displayed and interacted with system users for downstream tasks. Hence, designing an approach that can locate the copied segments is preferred and has already attracted lots of attentions in recent works [7][8][9][10][11].…”

“…Meanwhile, existing evaluation protocols for segmentlevel video copy detection exhibit an obvious drawback that most of them utilize ground-truth copied segments as queries rather than the entire videos [7,8,11]. This is unpractical for real copy detection scenario where it is hard to know a priori that which part of a video will be pirated.…”

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

“…The task of Image-to-Video Retrieval (IVR) [191]- [194] localizes video segments that contain similar activity as in a query image. Similarly, given a query video and a reference video, video re-localization (VRL) [195]- [198] localizes a segment in the reference video that semantically corresponds to the query video. Conceptually, the query is in the form of audio in AVEL, appearance vision in IVR, and motion vision in VRL, respectively.…”