Adaptive Focus for Efficient Video Recognition

Abstract

In this paper, we explore the spatial redundancy in video recognition with the aim to improve the computational efficiency. It is observed that the most informative region in each frame of a video is usually a small image patch, which shifts smoothly across frames. Therefore, we model the patch localization problem as a sequential decision task, and propose a reinforcement learning based approach for efficient spatially adaptive video recognition (AdaFocus). In specific, a light-weighted ConvNet is first adopted to quickly process the full video sequence, whose features are used by a recurrent policy network to localize the most task-relevant regions. Then the selected patches are inferred by a highcapacity network for the final prediction. During offline inference, once the informative patch sequence has been generated, the bulk of computation can be done in parallel, and is efficient on modern GPU devices. In addition, we demonstrate that the proposed method can be easily extended by further considering the temporal redundancy, e.g., dynamically skipping less valuable frames. Extensive experiments on five benchmark datasets, i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2, demonstrate that our method is significantly more efficient than the competitive baselines. Code will be available at https:// github.com/blackfeather-wang/AdaFocus.

Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection

“…The training in AR-Net is simplified using the Gumbel-Softmax trick. Later, this idea was extended to adaptively select a proper modality [20] or patches [42]. Our approach is motivated by these prior works to apply a similar framework to adaptive computation on deep learn-based VIO for the first time.…”

MAR: Masked Autoencoders for Efficient Action Recognition

“…Reducing spatio-temporal redundancy for efficient video analysis has recently been a popular research topic. The mainstream approaches mostly train an additional lightweight network to achieve: (i) adaptive frame selection [12]- [14], [16], [44], i.e., dynamically determining the relevant frames for the recognition networks; (ii) adaptive frame resolution [12], i.e., learning an optimal resolution for each frame online; (iii) early stopping [45], i.e., terminating the inference process before observing all frames; (iv) adaptive spatio-temporal regions [10], [11], i.e., localizing the most task-relevant spatiotemporal regions; (v) adaptive network architectures [15], [16], [46], i.e., adjusting the network architecture to save computation on less informative features. Another line is to manually define low redundant sampling rules, such as MGSampler [47], which selects frames containing rich motion information by the cumulative motion distribution.…”

“…Although this yields decent performances, the computation over full videos is highly redundant due to the excessive and widely present spatio-temporal redundancy of visual information [9]- [13] in videos. In light of this, a branch of previous works has proposed to reduce the spatiotemporal redundancy by training an additional model to focus on relevant frames [12]- [17] or spatio-temporal regions [10], [11], which can significantly reduce the computation cost. However, they mostly require complicated operations, such as reinforcement learning and multi-stage training.…”

E^2TAD: An Energy-Efficient Tracking-based Action Detector

“…For example, a dynamic network spends less computation on easy samples or less informative spatial areas/temporal locations of an input. For image [32,60] or video-related [30,59] tasks, sample-wise, spatial-wise, or temporal-wise adaptive inference could be conducted by formulating the recognition or detection task as a sequential decision problem and allowing early exiting during inference.…”