Spatio-Temporal Frames in a Bag-of-Visual-Features Approach for Human Actions Recognition

Abstract

Abstract-The recognition of human actions from videos has several interesting and important applications, and a vast amount of different approaches has been proposed for this task in different settings. Such approaches can be broadly categorized in model-based and model-free. Typically, model-based approaches work only in very constrained settings, and because of that, a number of model-free approaches appeared in the last years. Among them, those based in bag-of-visual-features (BoVF) have been proving to be the most consistently successful, being used by several independent authors.For videos to be represented by BoVFs, though, an important issue that arises is how to represent dynamic information. Most existing proposals consider the video as a spatio-temporal volume and then describe "volumetric patches" around 3D interest points. In this work, we propose to build a BoVF representation for videos by collecting 2D interest points directly. The basic idea is to gather such points not only from the traditional frames (xy planes), but also from those planes along the time axis, which we call the spatio-temporal frames. Our assumption is that such features are able to capture dynamic information from the videos, and are therefore well-suited to recognize human actions from them, without the need of 3D extensions for the descriptors. In our experiments, this approach achieved stateof-the-art recognition rates on a well-known human actions database, even when compared to more sophisticated schemes.

Human action recognition using simple geometric features and a finite state machine

“…The use of space-time features is presented in the work [7]. The Bag of Visual Features (BoVF) method is used, which consists of the following steps: characteristic points detection and description (SIFT algorithm), creating a dictionary of the extracted features (using clustering), assigning each detected point to a word from the dictionary (using the smallest distance criterion) and creating a histogram of used "visual words".…”

Spatio-temporal SIFT and Its Application to Human Action Classification

“…A pixel Gi(t) at level Gi maps to a pixel G0(2 i t). For example, G1(3) mapped to G0 (6) and G2(1) mapped to G0(4).…”

“…The assumption here is that spatio-temporal events can be described by common interest points between the spatial axis (appearance information) and the temporal axis (motion information). Lopes et al presented an approach to forming a spatio-temporal volume by stacking a set of frames from a video signal [6]. There are three directions to slice this volume into planes, as illustrated in Figure 3.…”

“…The second one was presented by Lopes et al for human action recognition application [6]. They collected 2D SIFT and 2D SURF (speeded up robust features) interest points on the xy plane along the spatial domain and on the xt and yt planes along the spatio-temporal domain.…”

Dynamic Spatio-Temporal Bag of Expressions (D-STBoE) Model for Human Action Recognition

“…However, there are still some unsolved issues such as background clutter, viewpoint variation, illumination change and class variability [2]. Recently, significant progress has been demonstrated with spatio-temporal feature representation along with variations of the most popular and widely used bag of visual words approaches (BoVW) [3], which have the ability to handle viewpoint independence, occlusion and scale invariance [4,5]. Therefore, there has been a growing interest in exploring the potential of possible variants of the classical BoVW approach, which characterizes actions using a histogram of feature occurrence after clustering [6].…”