Efficient Action Localization with Approximately Normalized Fisher Vectors

Abstract

International audienceThe Fisher vector (FV) representation is a high-dimensional extension of the popular bag-of-word representation. Transformation of the FV by power and L2 normalizations has been shown to significantly improve its performance. With these normalizations included, this representation has yielded state-of-the-art results for a wide number of image and video classification and retrieval tasks. The normalizations, however, render the representation non-additive over local descriptors. Combined with its high dimensionality, this makes the FV computationally very expensive for the purpose of localization tasks. In this paper we, first, present approximations to both these normalizations, which yield significant improvements in the memory requirements and computational costs of the FV when used for localization. Second, we show how these approximations can be used to define upper-bounds on the score function that can be efficiently evaluated, which paves the way for the use of branch-and-bound search as an alternative to exhaustive scanning window search. We present experimental evaluation results on classification and temporal localization of actions in videos. These show that the proposed approximations lead to speed-ups of at least one order of magnitude, while maintaining state-of-the-art action localization performance

DAPs: Deep Action Proposals for Action Understanding

“…We attribute this improvement to the fact that our approach scans the video in a much more efficient way. We obtain a similar performance to Caba Heilbron et al [4] and Oneata et al [25]. This result is encouraging given that our detection pipeline operates at a much faster rate of 134 FPS.…”

“…Action Detection: In contrast to object detection methods, the dominant approach for action detection is still to use a sliding window approach [26,18,12] combined with action classifiers trained on multiple features [2,9,33]. Previous approaches have reduced the computational overhead of sliding window search by using branch-and-bound techniques [5,27] and exploiting some characteristics of the visual descriptors. In contrast, our model efficiently reduces the number of evaluated windows by encoding a sequence of visual descriptors.…”

Modeling spatio-temporal layout with Lie Algebrized Gaussians for action recognition

“…Furthermore, only linear classifier is required by FV, which is a huge advantage in large-scale problems. Therefore, FV and its simplified non-probabilistic version VLAD become more popular in action recognition [3,10,12,21,22,24,34,36].…”

A novel hierarchical Bag-of-Words model for compact action representation

“…Among them, VLAD and FV show outstanding performances for human action recognition [5][24] [25] [27][28] [29]. Compared with BoW in Figure 1, VLAD records the 1st-order difference between local features and codewords, i.e., the residual vectors generated by hard assignment.…”