Comparison of Visual Features and Fusion Techniques in Automatic Detection of Concepts from News Video

Abstract

This study describes experiments on automatic detection of semantic concepts, which are textual descriptions about the digital video content. The concepts can be further used in content-based categorization and access of digital video repositories. Temporal Gradient Correlograms, Temporal Color Correlograms and Motion Activity low-level features are extracted from the dynamic visual content of a video shot. Semantic concepts are detected with an expeditious method that is based on the selection of small positive example sets and computational low-level feature similarities between video shots. Detectors using several feature and fusion operator configurations are tested in 60-hour news video database from TRECVID 2003 benchmark. Results show that the feature fusion based on ranked lists gives better detection performance than fusion of normalized low-level feature spaces distances. Best performance was obtained by pre-validating the configurations of features and rank fusion operators. Results also show that minimum rank fusion of temporal color and structure provides comparable performance.

Multimodal indexing based on semantic cohesion for image retrieval

“…Westerveld and van Gemert adopted a similar approach, although they applied dimensionality reduction techniques to the heterogeneous vectors for compressing the multimodal information (Westerveld 2000;van Gemert 2003). Similar approaches are reported in (Rautiainen et al 2004;Rautiainen and Seppdnen 2005;Snoek et al 2005). Compared to LF, EF can be more efficient as a single retrieval stage is performed, however, the dimensionality in which EF methods may work can be huge.…”

“…Usually a single method is used per modality (Peinado et al 2005;Izquierdo-Beviá et al 2005;Besancon and Millet 2006;Chang and Chen 2006;Rautiainen et al 2004), although the use of multiple and heterogeneous techniques has been also studied (Escalante et al 2008b). The EF formulation, on the other hand, consists of merging the vectors corresponding to textual and visual information beforehand and then using a straight retrieval technique (Rautiainen et al 2004;Rautiainen and Seppdnen 2005;Snoek et al 2005;Westerveld 2000;van Gemert 2003;). In its basic form, EF consists of concatenating the vectors of textual and visual features.…”

“…Feature vectors have been usually combined through late fusion (LF) or early fusion (EF) techniques. The LF approach to multimedia image retrieval consists of running several unimodal retrieval methods (either textual or visual) and combining their outputs to obtain a single list of ranked images (documents) per query (Westerveld 2004;Escalante et al 2008b;Peinado et al 2005;Izquierdo-Beviá et al 2005;Besancon and Millet 2006;Inf Retrieval (2012) 15:1-32 5 Chang and Chen 2006;Rautiainen and Seppdnen 2005;Rautiainen et al 2004;Snoek et al 2005). Because of its simplicity and its effectiveness, LF is one of the most used techniques for information fusion in general (Shu and Taska 2005), although a disadvantage of this method is that it may be inefficient as several retrieval methods must be run for each query.…”

Low-level feature fusion models for soccer scene classification

“…We notice that the coder neural-network obtains superior scores to those obtained by the other systems (1,2,3,4) for all semantic concepts. This supports further the importance of feature fusion.…”

“…Many application domains making use of video data are available: Security, digital library, interactive TV, etc... Many of those rely on video content analysis and in particular video shot classification [1,2].…”