Detecting Semantic Concepts from Video Using Temporal Gradients and Audio Classification

Abstract

Abstract. In this paper we describe new methods to detect semantic concepts from digital video based on audible and visual content. Temporal Gradient Correlogram captures temporal correlations of gradient edge directions from sampled shot frames. Power-related physical features are extracted from short audio samples in video shots. Video shots containing people, cityscape, landscape, speech or instrumental sound are detected with trained self-organized maps and kNN classification results of audio samples. Test runs and evaluations in TREC 2002 Video Track show consistent performance for Temporal Gradient Correlogram and state-of-the-art precision in audio-based instrumental sound detection.

Comparison of Visual Features and Fusion Techniques in Automatic Detection of Concepts from News Video

“…In [6] we observed that several concepts coexist and correlate in a video, which is not suitable for multi-class classifiers. Our approach is to have several simplified concept detectors that are trained using small sets of positive example shots, each propagating labels to their nearest neighbors in selected feature spaces.…”

“…TGC, initially used in the detector experiments in [6], describes spatial correlation of edge orientations in an autocorrelogram. The feature is computed from the 20 temporally sampled video frames in a shot.…”

“…However, training of multiple classifiers for large concept lexicon can be tedious. A fast and simple method to build concept detectors was introduced in [6], where detectors were trained by selecting only small sets of positive examples for every concept.…”

“…This paper presents extended experiments with visual detectors for 12 semantic concepts from TRECVID 2003 semantic feature detection task [13]. The detectors use low-level visual features that measure video motion activity and spatial correlations of image gradients and colors.…”

“…The detectors use low-level visual features that measure video motion activity and spatial correlations of image gradients and colors. In comparison to prior research on visual detectors [13] [6] this paper reports experiments with broader sets of concepts, low-level features, fusion techniques, training set sizes and larger test database. Section 2 describes selected low-level features and the fusion operations used in concept detectors.…”