Dense saliency-based spatiotemporal feature points for action recognition

Abstract

Several spatiotemporal feature point detectors have been recently used in video analysis for action recognition. Feature points are detected using a number of measures, namely saliency, cornerness, periodicity, motion activity etc. Each of these measures is usually intensity-based and provides a different trade-off between density and informativeness. In this paper, we use saliency for feature point detection in videos and incorporate color and motion apart from intensity. Our method uses a multi-scale volumetric representation of the video and involves spatiotemporal operations at the voxel level. Saliency is computed by a global minimization process constrained by pure volumetric constraints, each of them being related to an informative visual aspect, namely spatial proximity, scale and feature similarity (intensity, color, motion). Points are selected as the extrema of the saliency response and prove to balance well between density and informativeness. We provide an intuitive view of the detected points and visual comparisons against state-of-the-art space-time detectors. Our detector outperforms them on the KTH dataset using Nearest-Neighbor classifiers and ranks among the top using different classification frameworks. Statistics and comparisons are also performed on the more difficult Hollywood Human Actions (HOHA) dataset increasing the performance compared to current published results.

Detecting Human Action as the Spatio-Temporal Tube of Maximum Mutual Information

“…Our method achieves 94.4% and outperforms the approaches based on STIPs [2], [8], [9], [33], [34]. This validates the superiority of feature trajectories over STIPs in discriminating human actions.…”

Human action recognition with Optimized Video Densely Sampling

“…Among these methods, the human action recognition based on the bag of words (BoW) model has achieved satisfactory results in many tasks and drawn much attention. The this framework, and several STIP detectors have been introduced [6, 7, 8,9,10,11].…”

“…The STIP detector in [6] usually suffers from sparse detection results. Later, several methods for detecting STIPs have been reported [7, 8,9,10,11]. Dollr et al…”

“…It is a spatiotemporal extension of the Hessian saliency measure, previously applied for object detection. Oikonomopoulos et al [9] proposed a spatio-temporal extension of the salient region detector, and a dense version for this STIP was proposed by Rapantzikos et al [10]. Yu and Kim [11] proposed a STIPs detector which they call V-FAST (Video FAST) by extending the FAST corners [19] into a spatiotemporal domain.…”

“…Many STIP detectors have been proposed using many models [6,7,8,9,10,11]. STIP captures characteristic shape and motion in video and provides relatively independent representation of events with respect to their spatiotemporal shifts and scales invariant as well as background clutter and multiple motions in the scene.…”