Dominant spatio-temporal modulations and energy tracking in videos: Application to interest point detection for action recognition

Abstract

The presence of multiband amplitude and frequency modulations (AM-FM) in wideband signals, such as textured images or speech, has led to the development of efficient multicomponent modulation models for low-level image and sound analysis. Moreover, compact yet descriptive representations have emerged by tracking, through non-linear energy operators, the dominant model components across time, space or frequency. In this paper, we propose a generalization of such approaches in the 3D spatio-temporal domain and explore the potential of incorporating the Dominant Component Analysis scheme for interest point detection and human action recognition in videos. Within this framework, actions are implicitly considered as manifestations of spatio-temporal oscillations in the dynamic visual stream. Multiband filtering and energy operators are applied to track the source energy in both spatial and temporal frequency bands. A new measure for extracting keypoint locations is formulated as the temporal dominant energy computed over the spatial dominant components, in terms of their modulation energy, of input video frames. Theoretical formulation is supported by evaluation and comparisons in human action classification, which demonstrate the potential of the proposed spatio-temporal detector.Index Terms-Human action recognition in videos, spatiotemporal interest point detectors, multiband filtering, multicomponent AM-FM models, dominant component analysis

A perceptually based spatio-temporal computational framework for visual saliency estimation

“…1 Dominant component analysis on the outputs of Gabor filterbanks has been used for 2D texture analysis and segmentation in [91,90,92] and for spatio-temporal action classification in [93,94]. It may include additional steps of demodulation.…”

Advances on action recognition in videos using an interest point detector based on multiband spatio-temporal energies

“…1, the energy outputs of all 400 filters are handled by some operator in order to obtain the final energy map of each video. We used some ideas from Dominant Energy Analysis (DEA), as in [12], where the energy of the most dominant channel is considered as the energy value in each voxel:…”

“…Video representations in terms of such features exhibit efficiency in distinguishing among action classes, while bypassing the need for precise background subtraction or tracking. Local image and video features have been successfully used for many tasks such as object and scene recognition [23] as well as human action recognition [9,12,24,32]. Local spatio-temporal features are able to capture characteristic shape and motion in video.…”

“…Georgakis et al [12] introduced the DCA3D detector, which is based on multichannel filtering via Gabor filters and Dominant Component Analysis. First, the filtering process takes place in the 2 spatial dimensions and then the dominant component volume is filtered by temporal filters to lead to the final energy volume and the selection of interest points.Wang et al [35] introduced dense trajectories and motion boundary histograms to deal with the task of action recognition.…”

Dynamic Behavior Analysis via Structured Rank Minimization

“…Traditionally, research in behavior and affect analysis has focused on recognizing behavioral cues such as smiles, head nods, and laughter (Déniz et al 2008;Kawato and Ohya 2000;Lockerd and Mueller 2002), pre-defined posed human actions (e.g., walking, running, and hand-clapping) (Dollár et al 2005;Niebles et al 2008;Georgakis et al 2012) or discrete, basic emotional states (e.g., happiness, sadness) (Pantic and Rothkrantz 2000;Cohen et al 2003;Littlewort et al 2006) mainly from posed data acquired in laboratory settings. However, these models are deemed unrealistic as they are unable to capture the temporal evolution of non-basic, possibly atypical, behaviors and subtle affective states exhibited by humans in naturalistic settings.…”