Video Similarity and Alignment Learning on Partial Video Copy Detection

Abstract

Existing video copy detection methods generally measure video similarity based on spatial similarities between key frames, neglecting the latent similarity in temporal dimension, so that the video similarity is biased towards spatial information. There are methods modeling unified video similarity in an end-to-end way, but losing detailed partial alignment information, which causes the incapability of copy segments localization. To address the above issues, we propose the Video Similarity and Alignment Learning (VSAL) approach, which jointly models spatial similarity, temporal similarity and partial alignment. To mitigate the spatial similarity bias, we model the temporal similarity as the mask map predicted from frame-level spatial similarity, where each element indicates the probability of frame pair lying right on the partial alignments. To further localize partial copies, the step map is learned from the spatial similarity where the elements indicate extending directions of the current partial alignments on the spatial-temporal similarity map. Obtained from the mask map, the start points extend out into partial optimal alignments following instructions of the step map. With the similarity and alignment learning strategy, VSAL achieves the state-of-the-art 𝐹 1 -score on VCDB core dataset. Furthermore, we construct a new benchmark of partial video copy detection and localization by adding new segment-level annotations for FIVR-200k dataset, where VSAL also achieves the best performance, verifying its effectiveness in more challenging situations. Our project is publicly available at https://pvcd-vsal.github.io/vsal/. CCS CONCEPTS• Computing methodologies → Matching; Visual content-based indexing and retrieval.

A Large-scale TV Dataset for Partial Video Copy Detection

“…We have also removed the optimization components for a strongest accuracy. The F 1 score has been used as it is common to characterize the PVCD methods [9,4,19,6,5,15].…”

“…It is a well-known topic in the computer vision field [12]. The recent works investigate the detection methods robust to the spatial & temporal deformations [6,5,15] or real-time [4,13,19]. A key aspect for any computer vision task is to design public datasets for performance evaluation.…”

A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detection

“…Previous segment-level evaluation metrics are introduced with MUSCLE-VCD [15] and VCDB datasets [11]. Most of recent research works [7][8][9] adopt segment precision and recall defined in VCDB as follows:…”

“…In most cases, video-level copy detection results alone are not sufficient as the detected videos are usually displayed and interacted with system users for downstream tasks. Hence, designing an approach that can locate the copied segments is preferred and has already attracted lots of attentions in recent works [7][8][9][10][11].…”

STAR-GNN: Spatial-Temporal Video Representation for Content-based Retrieval

“…Kordopatis et al [10] calculate video-to-video similarity by refined frame-to-frame similarity matrices. Han et.al [11] modelled the video similarity as the mask map predicted from frame-level spatial similarity.…”