Video Object Detection with an Aligned Spatial-Temporal Memory

Xiao, Fanyi; Lee, Yong Jae

doi:10.1007/978-3-030-01237-3_30

Cited by 181 publications

(162 citation statements)

References 50 publications

Supporting

Mentioning

161

Contrasting

Order By: Relevance

“…Sampling Strategies for Feature Aggregation Frame sampling strategy matters for video detection. As previous works [33,36] pointed out, using more frames in feature aggregation during testing yields better results. Besides, [33] samples frames with a uniform stride during testing to improve the performance.…”

Section: Effectiveness Of Selsamentioning

confidence: 70%

“…STMN [33] used a Spatial-Temporal Memory module as the recurrent operation to pass the information through a video. Unlike [33], our method does not need to pass information using memory modules in temporal order. We form clusters and aggregate features in a multi-shot view to capture the rich information of videos instead.…”

Section: Object Detection In Videosmentioning

confidence: 99%

“…One advantage of our method is that it does not rely on post-processing methods (e.g Seq-NMS) to incorporate the full-sequence level information. Nearly all the state-of-theart video detection systems [36,29,6,1,33] adopted postprocessing methods which gives huge gains in performance. To illustrate that our method has already captured the fullsequence level information, we further apply the Seq-NMS post-processing upon our method.…”

Section: Video-level Post-processing Techniquesmentioning

confidence: 99%

“…FGFA* [36] ResNet-101 78.4 MANet* [29] 80.3 ST-Lattice* [1] 79.6 D&T* [6] 79.8 STMN*+ [33] 80 The middle part of Table 3 shows the comparison with methods that utilize sequence-level post-processing techniques. FGFA*, MANet* and STMN*+ [33] use Seq-NMS, while D&T* [6], ST-Lattice* [1] utilize tubelet rescoring. Our method, by using Seq-NMS as the post-processing method, achieves 80.54 mAP, which is slightly better than the previous state-of-the-art method STMN*+.…”

Section: (And T Loss)mentioning

confidence: 99%

See 3 more Smart Citations

Sequence Level Semantics Aggregation for Video Object Detection

Chen

Wang³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

189

185

View full text Add to dashboard Cite

Video objection detection (VID) has been a rising research direction in recent years. A central issue of VID is the appearance degradation of video frames caused by fast motion. This problem is essentially ill-posed for a single frame. Therefore, aggregating features from other frames becomes a natural choice. Existing methods rely heavily on optical flow or recurrent neural networks for feature aggregation. However, these methods emphasize more on the temporally nearby frames. In this work, we argue that aggregating features in the full-sequence level will lead to more discriminative and robust features for video object detection. To achieve this goal, we devise a novel Sequence Level Semantics Aggregation (SELSA) module. We further demonstrate the close relationship between the proposed method and the classic spectral clustering method, providing a novel view for understanding the VID problem. We test the proposed method on the ImageNet VID and the EPIC KITCHENS dataset and achieve new state-of-theart results. Our method does not need complicated postprocessing methods such as Seq-NMS or Tubelet rescoring, which keeps the pipeline simple and clean.

show abstract

Section: Effectiveness Of Selsamentioning

confidence: 70%

Section: Object Detection In Videosmentioning

confidence: 99%

Section: Video-level Post-processing Techniquesmentioning

confidence: 99%

Section: (And T Loss)mentioning

confidence: 99%

See 2 more Smart Citations

Sequence Level Semantics Aggregation for Video Object Detection

Chen

Wang³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

189

185

View full text Add to dashboard Cite

show abstract

“…Generalizing still image detectors to video domain is not trivial due to the spatial and temporal complex variations existed in videos, not to mention that the object appearances in some frames may be deteriorated by motion blur or occlusion. One common solution to amend this problem is feature aggregation [1,29,49,53,54,55] that enhances per-frame features by aggregating the features of nearby frames. Specifically, FGFA [54] utilizes the optical flow from FlowNet [7] to guide the pixel-level motion compensation on feature maps of adjacent frames for feature aggregation.…”

Section: Related Workmentioning

confidence: 99%

Relation Distillation Networks for Video Object Detection

Deng

Pan

Yao

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

197

152

View full text Add to dashboard Cite

It has been well recognized that modeling object-toobject relations would be helpful for object detection. Nevertheless, the problem is not trivial especially when exploring the interactions between objects to boost video object detectors. The difficulty originates from the aspect that reliable object relations in a video should depend on not only the objects in the present frame but also all the supportive objects extracted over a long range span of the video. In this paper, we introduce a new design to capture the interactions across the objects in spatio-temporal context. Specifically, we present Relation Distillation Networks (RDN) -a new architecture that novelly aggregates and propagates object relation to augment object features for detection. Technically, object proposals are first generated via Region Proposal Networks (RPN). RDN then, on one hand, models object relation via multi-stage reasoning, and on the other, progressively distills relation through refining supportive object proposals with high objectness scores in a cascaded manner. The learnt relation verifies the efficacy on both improving object detection in each frame and box linking across frames. Extensive experiments are conducted on Im-ageNet VID dataset, and superior results are reported when comparing to state-of-the-art methods. More remarkably, our RDN achieves 81.8% and 83.2% mAP with ResNet-101 and ResNeXt-101, respectively. When further equipped with linking and rescoring, we obtain to-date the best reported mAP of 83.8% and 84.7%.

show abstract

Spatio‐temporal feature fusion based correlative binary relevance for visual object detection

Amaresh

Chitrakala

2021

Concurrency and Computation

View full text Add to dashboard Cite

Object detection in video is an emerging and challenging research topic in Computer Vision. Recently, deep learning based approaches achieves greater results in object detection. The major issues in detecting multiple objects from the videos are exploiting low‐level visual concepts and temporal information and also exploring the correlation between the objects in the videos. Therefore, to address the above mentioned issues, the spatio‐temporal feature fusion based correlative binary relevance (STFF‐CBR) classifier is proposed to generate a rich vector representation and exploit the label correlation for the object detection in videos. In this article, first, the spatio‐temporal feature fusion (STFF) is proposed to exploit the low‐level visual concepts and temporal information in the videos which significantly improves the object detection performance. Second, correlative binary relevance (CBR) classification approach is proposed to exploit the dependencies between the labels in the video using the nearest neighbor based label dependency graph (LDG). Additionally, feed forward neural network (FFNN) classifier is utilized to increase the classification accuracy of the CBR method. Experimental evaluation shows that the STFF‐CBR classifier model achieves better performance for object detection problem in video against the state‐of‐the‐art methods.

show abstract

Video Object Detection with an Aligned Spatial-Temporal Memory

Cited by 181 publications

References 50 publications

Sequence Level Semantics Aggregation for Video Object Detection

Sequence Level Semantics Aggregation for Video Object Detection

Relation Distillation Networks for Video Object Detection

Spatio‐temporal feature fusion based correlative binary relevance for visual object detection

Contact Info

Product

Resources

About