Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video

Abstract

Current approaches for activity recognition often ignore constraints on computational resources: 1) they rely on extensive feature computation to obtain rich descriptors on all frames, and 2) they assume batch-mode access to the entire test video at once. We propose a new active approach to activity recognition that prioritizes "what to compute when" in order to make timely predictions. The main idea is to learn a policy that dynamically schedules the sequence of features to compute on selected frames of a given test video. In contrast to traditional static feature selection, our approach continually re-prioritizes computation based on the accumulated history of observations and accounts for the transience of those observations in ongoing video. We develop variants to handle both the batch and streaming settings. On two challenging datasets, our method provides significantly better accuracy than alternative techniques for a wide range of computational budgets.

AdaFrame: Adaptive Frame Selection for Fast Video Recognition

“…Most existing work focuses on extending 2D convolution to the video domain and modeling motion information in videos [19,29,28,35,24]. Only a few methods consider efficient video classifica-tion [38,31,40,20,10]. However, these approaches perform mean-pooling of scores/features from multiple frames, either uniformly sampled or decided by an agent, to classify a video clip.…”

BlockDrop: Dynamic Inference Paths in Residual Networks

“…In contrast, our policy network makes all routing decisions in a single step, resulting in lower overhead cost for the routing itself and thus larger computational savings. Reinforcement learning has also been applied for dynamic feature prioritization in images [26] and video [45,56], actively deciding which frames or image regions to visit next. These techniques could be used in tandem with our approach.…”

TAB: Temporally aggregated bag-of-discriminant-words for temporal action proposals

“…We found many research works addressing the issue of temporal action localization (Shou et al, 2016;Caba Heilbron et al, 2016;Escorcia et al, 2016;Karaman et al, 2014;Buch et al, 2017;Oneata et al, 2014;Gao et al, 2017;Su and Grauman, 2016;Sun et al, 2015;Wang et al, 2014;Yuan et al, 2015;Tran et al, 2015;Singh et al, 2016;Duchenne et al, 2009). A traditional way of performing temporal action detection is to densely apply action classifiers in a sliding window fashion (Duchenne et al, 2009).…”