Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words

Abstract

DESCRIPTIONImagine a video taken on a sunny beach, can a computer automatically tell what is happening in the scene? Can it identify different human activities in the video, such as water surfing, people walking and lying on the beach? To automatically classify or localize different actions in video sequences is very useful for a variety of tasks, such as video surveillance, objectlevel video summarization, video indexing, digital library organization, etc. However, it remains a challenging task for computers to achieve robust action recognition due to cluttered background, camera motion, occlusion, and geometric and photometric variances of objects. For example, in a live video of a skating competition, the skater moves rapidly across the rink, and the camera also moves to follow the skater. With moving camera, non-stationary background, and moving target, few vision algorithms could identify, categorize and localize such motions well. In addition, the challenge is even greater when there are multiple activities in a complex video sequence ( Figure 1).We present a video demo for our novel unsupervised learning method for human action categories [1]. A video sequence is represented as a collection of spatial-temporal words by extracting space-time interest points. The algorithm learns the probability distributions of the spatial-temporal words and intermediate topics corresponding to human action categories automatically using a probabilistic Latent Semantic Analysis (pLSA) model [4]. The learned model is then used for human action categorization and localization in a novel video, by maximizing the posterior of action category (topic) distributions. The contributions of this work are as follows:• Unsupervised learning of actions using 'video words' representation. We deploy a pLSA model with 'bag of video words' representation for video analysis; • Multiple action localization and categorization. Our approach is not only able to classify different actions, but also to localize different actions simultaneously in a novel and complex video sequence. We test our algorithm on two datasets: the KTH human action dataset [3] and a recent dataset of figure skating actions [2]. These datasets contain videos of cluttered background, moving camera, and multiple actions. Our results on action recognition performance are on par or slightly better than the best reported results in the literature. In addition, our algorithm can recognize and localize multiple actions in long and complex video sequences containing multiple motions.

Action Recognition Using a Bio-Inspired Feedforward Spiking Network

“…For example, in Laptev et al (2007), the authors propose to use event-based local motion representations (here, spatial-temporal chunks of a video corresponding to 2D + t edges) and template matching. This idea extracting spatial-temporal features was proposed in several contributions such as Dollar et al (2005), and then Niebles et al (2006), Wong et al (2007), using the notion of cuboids. Another stream of approaches was inspired by the work by Serre (2006), first applied to object recognition Mutch and Lowe 2006) and then extended to action recognition (Sigala et al 2005;Jhuang et al 2007).…”

Task recognition from joint tracking data in an operational manufacturing cell

“…[Zhang and Parker (2011) use an unsupervised learning algorithm with Kinect data, but create their own segmented, scripted, laboratory data set.] On the other hand, some studies have used unsupervised or semisupervised algorithms and real-world data with other types of sensors (Krause et al 2003;Wang et al 2009;Stikic et al 2008;Niebles et al 2008;Mahdaviani and Choudhury 2008). Table 1 lists the attributes of several studies, including those cited in the previous paragraph, with a column of indicators for each attribute.…”

“…While we classify this as real-world data, it was not collected in a real-world setting, due to the intrusiveness of on-body sensors. Niebles et al (2008) used video segments of figure skaters. While the activities were not scripted, the authors did preselect video segments for x" indicates that the research includes the attribute analysis.…”

Trajectory Analysis and Semantic Region Modeling Using Nonparametric Hierarchical Bayesian Models

“…They were used to solve the problems of scene categorization (FeiFei and Perona 2005;Sudderth et al 2007), object recognition (Sivic et al 2005;Sudderth et al 2007;, human action recognition (Niebles et al 2006;Niebles and Fei-Fei 2007) and video analysis (Wang et al 2009). Fox and Willsky et al (2006) used Dirichlet process to solve the problem of data association for multi-target tracking in the presence of an unknown number of targets.…”