Spatio-temporal SIFT and Its Application to Human Action Classification

Abstract

Abstract. This paper presents a space-time extension of scale-invariant feature transform (SIFT) originally applied to the 2-dimensional (2D) volumetric images. Most of the previous extensions dealt with 3-dimensional (3D) spacial information using a combination of a 2D detector and a 3D descriptor for applications such as medical image analysis. In this work we build a spatio-temporal difference-of-Gaussian (DoG) pyramid to detect the local extrema, aiming at processing video streams. Interest points are extracted not only from the spatial plane (xy) but also from the planes along the time axis (xt and yt). The space-time extension was evaluated using the human action classification task. Experiments with the KTH and the UCF sports datasets show that the approach was able to produce results comparable to the state-of-the-arts.

Saliency guided local and global descriptors for effective action recognition

“…From a computational cost point of view, SGSH Method on KTH Accuracy (%) Al Ghamdi et al [27] 90.7 Liu et al [28] 91.3 Iosifidis et al [29] 92.1 Baumann et al [30] 92.1 Kläser [31] 92.6 Ji et al [32] 93.1 Wang et al [33] 94.2 Wu et al [34] 94.5 Raptis and Soatto [35] 94.8 Zhang et al [5] 94.8 Wang et al [21] 95.0 Yuan et al [4] 95.4 SGSH 97.2 Table 3 Recognition accuracy comparisons on the UCF-Sports dataset…”

Simplex-Based 3D Spatio-temporal Feature Description for Action Recognition

“…Another popular 3D feature description methodology is based on regular polyhedrons [1,7,11,13,25]. This technique approximates the orientation space by a regular polyhedron with congruent faces that are regular polygons, each of which serves as a bin.…”

“…• 3D spatio-temporal features computed in xyt spatiotemporal space using a temporal sequence of images, including 3D SIFT [18,23], ST-SIFT [1], HOG3D [13], CHOG3D [11], 3D optical flow [10], etc.…”

“…The compared methods generally follow similar experimental setups that are based on feature pooling, bag-of-features encoding and SVM-based classification. Following [1,4,7,11,27], we adopt a spatiotemporal pooling scheme that divides each support region into 4×4×3 cells to construct bag-of-features models.…”

“…To recognize unconstrained human actions, a large number of 3D local spatio-temporal features have been recently introduced that are computed in xyt (i.e., 2D spatial and 1D temporal) space [1,4,7,13,23]. Although orientation Figure 2: Issues of previous 3D feature description methodologies: Spherical coordinate based approaches suffer from the singularity issue (Figure 2a): bins at the poles (red triangle) are significantly smaller than bins around the equator (blue rectangle).…”