Spacetime Forests with Complementary Features for Dynamic Scene Recognition

Abstract

This paper presents spacetime forests defined over complementary spatial and temporal features for recognition of naturally occurring dynamic scenes. The approach improves on the previous state-of-the-art in both classification and execution rates. A particular improvement is with increased robustness to camera motion, where previous approaches have experienced difficulty. There are three key novelties in the approach. First, a novel spacetime descriptor is employed that exploits the complementary nature of spatial and temporal information, as inspired by previous research on the role of orientation features in scene classification. Second, a forest-based classifier is used to learn a multi-class representation of the feature distributions. Third, the video is processed in temporal slices with scale matched preferentially to scene dynamics over camera motion. Slicing allows for temporal alignment to be handled as latent information in the classifier and for efficient, incremental processing. The integrated approach is evaluated empirically on two publically available datasets to document its outstanding performance.

Bags of Spacetime Energies for Dynamic Scene Recognition

“…Local smoothing is also appropriate, because the responses are directly used for subsequent encoding. This is in contrast to previous work using similar oriented filter responses for dynamic scene recognition which immediately aggregated filter responses over some support region (e.g., [6,9]). Thus, for every spacetime location, x, the local oriented energy E(x; θ i , σ j ) measures the power of local oriented structure along each considered orientation θ i and scale σ j .…”

“…Various primitive features have been investigated for dynamic scenes, including, flow vectors [16], linear dynamical systems [22], chaotic invariants [22], spatiotemporal orientations [6,9] and slowly varying spatial orientations [24]. Here, systematic comparisons suggest that spatiotemporal orientations provide particularly strong primitives for dynamic scene recognition [6,9].…”

“…Here, systematic comparisons suggest that spatiotemporal orientations provide particularly strong primitives for dynamic scene recognition [6,9].…”

“…Distribution of spatiotemporal oriented energies for 16 frames of a sequence from the YUPENN dataset (a) [6]. In (b)-(k) the oriented energies, calculated by Gaussian derivative filtering, Previous evaluations [6,9,22] showed that integrating colour cues is useful for dynamic scene categorization. Colour information is incorporated in the present spacetime primitives via the addition of three smoothed colour measurements,…”

“…The dominant approach to pooling throughout the dynamic scene recognition literature is that of spatial pyramids [6,9,22,24], which employ predefined hierarchically gridded aggregation regions to capture coarse geometric layout in a multiresolution fashion. A limitation of this approach is that its predefined aggregation regions do not adapt to the dynamics of a video sequence, e.g., as regions of interest move to different relative positions with the passage of time and when camera and scene motion are confounded.…”