Flow-Guided Feature Aggregation for Video Object Detection

Abstract

Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flowguided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong single-frame baselines in ImageNet VID [33], especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The proposed method, together with Deep Feature Flow [49], powered the winning entry of ImageNet VID challenges 2017 1 . The code is available at https://github.com/msracver/ Flow-Guided-Feature-Aggregation. * This work is done when Xizhou Zhu and Yujie Wang are interns at Microsoft Research Asia 1 http://image-net.org/challenges/LSVRC/2017/ results 1

Relation Distillation Networks for Video Object Detection

“…Generalizing still image detectors to video domain is not trivial due to the spatial and temporal complex variations existed in videos, not to mention that the object appearances in some frames may be deteriorated by motion blur or occlusion. One common solution to amend this problem is feature aggregation [1,29,49,53,54,55] that enhances per-frame features by aggregating the features of nearby frames. Specifically, FGFA [54] utilizes the optical flow from FlowNet [7] to guide the pixel-level motion compensation on feature maps of adjacent frames for feature aggregation.…”

“…D&T [8] integrates a tracking formulation into R-FCN [5] to simultaneously perform object detection and across-frame track regression. [46] further extends FGFA [54] by calibrating the object features on box level to boost video object detection.…”

“…Given the fact that the ground truth of the official testing set are not publicly available, we follow the widely adopted setting as in [8,20,46,49,54,53] to report mean Average Precision (mAP) on validation set. Following the common protocols in [8,46,49,54], we utilize both ImageNet VID and ImageNet object detection (DET) dataset to train our RDN. Since the 30 classes in Im-ageNet VID are a subset of 200 classes in ImageNet DET dataset, we adopt the images from overlapped 30 classes in ImageNet DET for training.…”

“…Training and Inference. At training stage, we adopt the strategy of temporal dropout [54] to randomly select two support frames I t+τ1 and I t+τ2 (τ 1 , τ 2 ∈ [−T, T ]) from the adjacent frames {I τ } t+T τ =t−T . Accordingly, the whole RDN is optimized with both classification and regression losses over the relation-augmented reference proposals R r2 from advanced stage in an end-to-end manner.…”

Great Ape Detection in Challenging Jungle Camera Trap Footage via Attention-Based Spatial and Temporal Feature Blending

“…To address this, FGFA [29] performs optical flow guided spatial warping before aggregating features. The resulting features are subsequently fused temporally by weighted element-wise addition where weights are determined by the optical flow field.…”