A Dynamic Frame Selection Framework for Fast Video Recognition

Wu, Zuxuan; Li, Hengduo; Xiong, Caiming; Jiang, Yu‐Gang; Davis, Larry S.

doi:10.1109/tpami.2020.3029425

Cited by 34 publications

(32 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the limited space, we introduce them in Appendix A. Following the common practice [19,40,47,48,64,68,69], we evaluate the performance of different methods via mean average precision (mAP) and Top-1 accuracy (Top-1 Acc.) on ActivityNet/FCVID and other datasets, respectively.…”

Section: Methodsmentioning

confidence: 99%

“…Temporal redundancy. A popular approach for facilitating efficient video recognition is to reduce the temporal redundancy in videos [19,21,35,36,47,57,67,68,72]. Since not all frames are equally important for a given task, the model should ideally allocate less computation on less informative frames [24].…”

Section: Related Workmentioning

confidence: 99%

“…Sampler [36], ListenToLook [19], AR-Net [47], AdaFrame [68], AdaFuse [48], VideoIQ [57], Dynamic-STE [35] and FrameExit [21]. An introduction is given in Appendix A.…”

Section: Comparisons With State-of-the-art Baselinesmentioning

confidence: 99%

“…Several algorithms have been proposed to reduce the temporal redundancy of videos [19,21,35,36,47,57,67,68,72] by allocating the majority of computation to the most task-relevant video frames rather than all. Orthogonal to these approaches, the recently proposed adaptive focus network (AdaFocus) [64] reveals that reducing the spatial redundancy in video analysis yields promising results for efficient video recognition.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

Wang¹,

Yang²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent works have shown that the computational efficiency of video recognition can be significantly improved by reducing the spatial redundancy. As a representative work, the adaptive focus method (AdaFocus) has achieved a favorable trade-off between accuracy and inference speed by dynamically identifying and attending to the informative regions in each video frame. However, AdaFocus requires a complicated three-stage training pipeline (involving reinforcement learning), leading to slow convergence and is unfriendly to practitioners. This work reformulates the training of AdaFocus as a simple one-stage algorithm by introducing a differentiable interpolation-based patch selection operation, enabling efficient end-to-end optimization. We further present an improved training scheme to address the issues introduced by the one-stage formulation, including the lack of supervision, input diversity and training stability. Moreover, a conditional-exit technique is proposed to perform temporal adaptive computation on top of AdaFocus without additional training. Extensive experiments on six benchmark datasets (i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2, and Jester) demonstrate that our model significantly outperforms the original AdaFocus and other competitive baselines, while being considerably more simple and efficient to train. Code is available at https://github.com/ LeapLabTHU/AdaFocusV2.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…Sampler [36], ListenToLook [19], AR-Net [47], AdaFrame [68], AdaFuse [48], VideoIQ [57], Dynamic-STE [35] and FrameExit [21]. An introduction is given in Appendix A.…”

Section: Comparisons With State-of-the-art Baselinesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

Wang¹,

Yang²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-agent has also been proposed in [5] to formulate the frame sampling as multiple parallel Markov decision processes. More recently, Wu et al [42] trained a policy network which contains a long short-term memory to provide the context information, it can interact with the video sequence and decide which frames to use dynamically. Besides, another group of methods focus on reducing spatial redundancy.…”

Section: B Reducing Spatial and Temporal Redundancymentioning

confidence: 99%

Action Keypoint Network for Efficient Video Recognition

Chen¹,

Han²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reducing redundancy is crucial for improving the efficiency of video recognition models. An effective approach is to select informative content from the holistic video, yielding a popular family of dynamic video recognition methods. However, existing dynamic methods focus on either temporal or spatial selection independently while neglecting a reality that the redundancies are usually spatial and temporal, simultaneously. Moreover, their selected content is usually cropped with fixed shapes (e.g., temporally-cropped frames, spatially-cropped patches), while the realistic distribution of informative content can be much more diverse. With these two insights, this paper proposes to integrate temporal and spatial selection into an Action Keypoint Network (AK-Net). From different frames and positions, AK-Net selects some informative points scattered in arbitrary-shaped regions as a set of "action keypoints" and then transforms the video recognition into point cloud classification. More concretely, AK-Net has two steps, i.e., the keypoint selection and the point cloud classification. First, it inputs the video into a baseline network and outputs a feature map from an intermediate layer. We view each pixel on this feature map as a spatialtemporal point and select some informative keypoints using selfattention. Second, AK-Net devises a ranking criterion to arrange the keypoints into an ordered 1D sequence. Since the video is represented with a 1D sequence after the specified layer, AK-Net transforms the subsequent layers into a point cloud classification sub-net by compacting the original 2D convolutional kernels into 1D kernels. Consequentially, AK-Net brings two-fold benefits for efficiency: The keypoint selection step collects informative content within arbitrary shapes and increases the efficiency for modeling spatial-temporal dependencies, while the point cloud classification step further reduces the computational cost by compacting the convolutional kernels. Experimental results show that AK-Net can consistently improve the efficiency and performance of baseline methods on several video recognition benchmarks.

show abstract

Fusing surveillance videos and three‐dimensional scene: A mixed reality system

Cui

Khan

et al. 2022

Computer Animation & Virtual

View full text Add to dashboard Cite

Augmented Virtual Environments (AVE) or Virtual-Reality Fusion systems fuse dynamic videos with static three-dimensional (3D) models of a virtual environment to provide an optimal solution for visualizing and understanding multichannel surveillance systems. However, texture distortion caused by viewpoint changes in such systems is a critical issue that needs to be addressed. To minimize texture fusion distortion, this paper presents a novel virtual environment system in two phases, offline and online phases, to dynamically fuse multiple surveillance videos with a virtual 3D scene. In the offline phase, a static virtual environment is obtained by performing a 3D photogrammetric reconstruction from the input images of the scene. In the online phase, the virtual environment is augmented by fusing multiple videos through two optional strategies. One strategy is to dynamically map images of different videos onto a 3D model of the virtual environment, and the other is to extract moving objects and represent them as billboards. The system can be used to visualize a 3D environment from any viewpoint augmented by real-time videos. Experiments and user studies in different scenarios demonstrate the superiority of our system.

show abstract

A Dynamic Frame Selection Framework for Fast Video Recognition

Cited by 34 publications

References 40 publications

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

Action Keypoint Network for Efficient Video Recognition

Fusing surveillance videos and three‐dimensional scene: A mixed reality system

Contact Info

Product

Resources

About