Recognition of visual activities and interactions by stochastic parsing

Abstract

ÐThis paper describes a probabilistic syntactic approach to the detection and recognition of temporally extended activities and interactions between multiple agents. The fundamental idea is to divide the recognition problem into two levels. The lower level detections are performed using standard independent probabilistic event detectors to propose candidate detections of low-level features. The outputs of these detectors provide the input stream for a stochastic context-free grammar parsing mechanism. The grammar and parser provide longer range temporal constraints, disambiguate uncertain low-level detections, and allow the inclusion of a priori knowledge about the structure of temporal events in a given domain. To achieve such a system we: 1) provide techniques for generating a discrete symbol stream from continuous low-level detectors; 2) extend stochastic context-free parsing to handle uncertainty in the input symbol stream; 3) augment a run-time parsing algorithm to enforce intersymbol constraints such as requiring temporal consistency between primitives; and 4) extend the consistency filtering to maintain consistent multiobject interactions. We develop a real-time system and demonstrate the approach in several experiments on gesture recognition and in video surveillance. In the surveillance application, we show how the system correctly interprets activities of multiple, interacting objects.

Tracking the Evolution of a Tennis Match Using Hidden Markov Models

“…In the work by Ivanov and Bobick [5], we can see that the authors introduced a framework by which they analyse each complex event into its constituent elementary actions; in one of the examples the authors have used, a gesture is broken down into simple hand trajectories, which can be tracked more successfully via HMMs. Then, they apply Stochastic Context-Free Grammars to infer the full gesture.…”

“…Hence, a great number of cognitive visions applications have deployed such tools in order to perform information extraction from video sequences. Some of those attempts would include those of [1] for automatic annotation of Formula 1 race programs, [2] for recognising various types of strokes from tennis players during a tennis match, [5] for the recognition of general hand gestures, [3] for the recognition of American Sign Language and [4] for general human pose estimation. Apparently, such pieces of work may prove to be extremely useful in our case as well, and they can certainly serve as a guideline of what is feasible in cognitive vision, and what issues in this area are still open.…”

Modeling and Inference with Relational Dynamic Bayesian Networks

“…In (Ivanov & Bobick, 2000), (Moore & Essa, 2002) and (Ryoo & Aggarwal, 2006) the interactions between different agents is recognized. These methods recognize single-actor, simple activities and organize the detected simple activities with a stochastic grammar free parsing to recognize complex activities.…”

A Comparison of HMMs and Dynamic Bayesian Networks for Recognizing Office Activities

“…Some of the earliest work was done by Starner and Pentland in [18] where they used HMMs for recognizing hand movements in American Sign Language and by Oliver et al [12] to recognize facial expressions. More complex models, such as Parameterized-HMMs [19], Entropic-HMMs [1], Variable-length HMMs [7], Coupled-HMMs [2], structured HMMs [17] and context-free grammars [8] have been used to recognize more complex activities such as the interaction between two people.…”