Improving bag-of-features action recognition with non-local cues

Abstract

Local space-time features have recently shown promising results within Bag-of-Features (BoF) approach to action recognition in video. Pure local features and descriptors, however, provide only limited discriminative power implying ambiguity among features and sub-optimal classification performance. In this work, we propose to disambiguate local space-time features and to improve action recognition by integrating additional nonlocal cues with BoF representation. For this purpose, we decompose video into region classes and augment local features with corresponding region-class labels. In particular, we investigate unsupervised and supervised video segmentation using (i) motion-based foreground segmentation, (ii) person detection, (iii) static action detection and (iv) object detection. While such segmentation methods might be imperfect, they provide complementary region-level information to local features. We demonstrate how this information can be integrated with BoF representations in a kernel-combination framework. We evaluate our method on the recent and challenging Hollywood-2 action dataset and demonstrate significant improvements.

Motion Based Foreground Detection and Poselet Motion Features for Action Recognition

“…Table 2 shows that our approach outperforms all other methods reported in literature so far. In particular, results are better than those of Ullah et al [7], who used a person detector, and Vig et al [14], who built upon saliency from an eye tracking system. A detailed analysis of the contribution of each set of features is presented in Table 3.…”

“…However, such failure cases are quite rare, and they are outnumbered by those cases where current person detectors fail. Hence, compared to [7], we achieve much better performance. We also compare to [14], where an eye tracking system was used to emphasize the part of the image that humans consider most important.…”

“…al. [26] 50.9% Ullah et al [7] 55.7% Wang et al [3] 58.3% Jiang et al [4] 60.3% Vig et al [14] 61.9% Jain et al [5] 62.5% Oneaţȃ et al [27] 63.3% Wang et al [6] 64.3% Our method 67.8% Table 2. Comparison of our results to the currently best results reported in literature.…”

“…In contrast to Ullah et al [7] and Prest et al [8], who suggested finding the actor and other persons in the video directly with a person detector, we propose to first detect the foreground items based on motion before running poselet detectors [12] to localize the (relevant) actor(s) in these foreground areas. While there has been much progress on person detectors in Fig.…”

“…State-of-the-art works rather rely on global feature aggregation that do not make explicit use of the notion of an actor [2][3][4][5][6]. Exceptions are Ullah et al [7], who run a person detector to find persons, and Prest et al [8], who even try to detect action specific objects, such as cups and cigarettes. Wang et.…”