A Two-View Concept Correlation Based Video Annotation Refinement

Abstract

Recently, concept correlation defining the relationship between concepts has been playing an important role in video annotation (or concept detection). To improve the annotation performance, this paper presents a two-view concept correlation based video annotation refinement, using data-specific spatial and temporal concept correlations. Specifically, instead of generic concept correlation within shots, the spatial view estimates a data-specific concept correlation for each shot, via introducing concept correlation bases to map low-level features to high-level concept distribution under the framework of sparse representation. On the other hand, beyond the temporal consistency of one concept, a richer temporal correlation between different concepts respectively locating in the current shot and its neighbors is utilized to adjust the detection scores. In the end, these two types of concept correlations are integrated into a probability calculation based framework to refine the initial results derived from multiple concept detectors. And the experiments conducted on TRECVID 2006-2008 datasets and comparison with existing works demonstrate its effectiveness.

Fine‐granularity semantic video annotation

“…Annotation refinement: accuracy of automatic concept detection suffers due to the high variability in the visual characteristics and, hence, it is a common practice to minimize the effect of the error by refining obtained annotations. Several works use a popular strategy called Content-Based Concept Fusion(CBCF) (Zhong & Miao, 2012) to achieve this. In its simplest form, CBCF uses some common concept coocurrence reference to evaluate the quality of candidate annotations within shots.…”

“…Therefore, approaches that use within-shot correlation as well as temporal correlation are adopted. For instance, authors of (Zhong & Miao, 2012) propose the following method. Given a shot x t | t = 1, .…”

“…To refine the shot annotations resulting from the classification, we use a modified version of the strategy prposed by authors of (Zhong & Miao, 2012). Temporal refinement term is calculated over a window of a 10 shots and spatial refinement is done using WordNet and DBPedia based voting.…”