Spatiotemporal Features for Action Recognition and Salient Event Detection

Abstract

Although the mechanisms of human visual understanding remain partially unclear, computational models inspired by existing knowledge on human vision have emerged and applied to several fields. In this paper, we propose a novel method to compute visual saliency from video sequences by counting in the actual spatiotemporal nature of the video. The visual input is represented by a volume in space-time and decomposed into a set of feature volumes in multiple resolutions. Feature competition is used to produce a saliency distribution of the input implemented by constrained minimization. The proposed constraints are inspired by and associated with the Gestalt laws. There are a number of contributions in this approach, namely extending existing visual feature models to a volumetric representation, allowing competition across features, scales and voxels, and formulating constraints in accordance with perceptual principles. The resulting saliency volume is used to detect prominent spatiotemporal regions and consequently applied to action recognition and perceptually salient event detection in video sequences. Comparisons against established methods on public datasets are given and reveal the potential of the proposed model. The experiments include three action recognition scenarios and salient temporal segment detection in a movie database annotated by humans.

Detection of activities and events without explicit categorization

“…Second, we demonstrated that our method detects wide spectra of events as something significantly different from 'normal' states and also identifies their time intervals (e.g., N h and M h in the hierarchy of Figure 4) which cannot be obtained in the reference method [10]. Moreover, the proposed algorithm require extremely small datasets (e.g., on the order of 10 to 50 frames) in the training phase, whereas it is, in general, necessary to prepare huge amount of video data for learning action category (e.g., normal states) in conventional methods [1,7,10,13,15,20]. For future work, we will extend the proposed framework to multi-category event detection with explicit categorization.…”

“…Most action (event) detection methods extract features such as optical flow based features [12,20], spatio-temporal features [5,[10][11][12][13][14][15]27], or static features including appearance, shape, and spatial relations of local features [8,18]. Some unsupervised approaches, after extraction of those primitive features, utilize codebook representation [2,13,19] effective for describing and discriminating various event categories.…”

RVM-based human action classification through Gabor and Haar feature extraction

“…To demonstrate generalisability, this method has been systematically tested on a variety of datasets and shown to be more effective and accurate for action recognition. Rapantzikos et al (2011) developed a method to compute visual saliency from video sequence by counting the actual spatio temporal nature of video. The visual input is represented by a volume in space-time.…”

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

“…across time or perceptual scene. During the last years, many computational frameworks have been proposed for attention and saliency modeling, since they play a significant role in various multimedia applications, such as action recognition [7][8][9], behavioral analysis, and movie summarization [10][11][12].…”