Rank Pooling for Action Recognition

Abstract

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data -e.g. how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features. *

Dynamic Image Networks for Action Recognition

“…1, the dynamic image reflects different parts of the video; for instance, the rails that appear as a secondary motion contributor are superimposed on top of the horses and the jockeys who are the main actors. Such observations were also made in [7].…”

“…For example, the authors of [17,29,29] have shown that local motion patterns in short frame sequences can capture very well the short temporal structures in actions. The rank pooling idea, on which our dynamic images are based, was proposed in [6,7] using hand-crafted representation of the frames, while in [5] authors increase the capacity of rank pooling using a hierarchical approach.…”

“…As a representative goal we consider the standard problem of recognizing human actions in short video sequences. Recent works such as [5,6,7,9,22] pointed out that long term dynamics and temporal patterns are a very important cues for the recognition of actions. However, representing complex long term dynamics is difficult, particularly if one seeks compact representations that can be processed efficiently.…”

“…1). The dynamic image is obtained as a ranking classifier that, similarly to [6,7], sorts video frames temporally; the difference is that we compute this classifier directly at the level of the image pixels instead of using an intermediate feature representation.…”

A Comprehensive Review on Handcrafted and Learning-Based Action Representation Approaches for Human Activity Recognition

“…This dataset is meant for evaluation of HAR algorithms in real life scenarios. Many researchers have evaluated their algorithms on this dataset, the best accuracy achieved so far is 75.2% in [169] using rank pooling and CNN. Some example frames from Hollywood2 dataset are shown in Figure 15.…”