SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping

Stone, Austin; Maurer, Daniel; Ayvaci, Alper; Angelova, Anelia; Jonschkowski, Rico

doi:10.1109/cvpr46437.2021.00388

Cited by 55 publications

(55 citation statements)

References 37 publications

(83 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SelFlow [23] and Autoflow [36] are two selfsupervised methods that generate synthetic annotations. SMURF [34] integrates a set of techniques to do selfsupervised learning on unannotated video frames and has achieved promising results.…”

Section: Related Workmentioning

confidence: 99%

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

Sui¹,

Li²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Optical flow estimation aims to find the 2D motion field by identifying corresponding pixels between two images. Despite the tremendous progress of deep learning-based optical flow methods, it remains a challenge to accurately estimate large displacements with motion blur. This is mainly because the correlation volume, the basis of pixel matching, is computed as the dot product of the convolutional features of the two images. The locality of convolutional features makes the computed correlations susceptible to various noises. On large displacements with motion blur, noisy correlations could cause severe errors in the estimated flow. To overcome this challenge, we propose a new architecture "CRoss-Attentional Flow Transformer" (CRAFT), aiming to revitalize the correlation volume computation. In CRAFT, a Semantic Smoothing Transformer layer transforms the features of one frame, making them more global and semantically stable. In addition, the dot-product correlations are replaced with transformer Cross-Frame Attention. This layer filters out feature noises through the Query and Key projections, and computes more accurate correlations. On Sintel (Final) and KITTI (foreground) benchmarks, CRAFT has achieved new state-of-the-art performance. Moreover, to test the robustness of different models on large motions, we designed an image shifting attack that shifts input images to generate large artificial motions. Under this attack, CRAFT performs much more robustly than two representative methods, RAFT and GMA. The code of CRAFT is is available at https://github.com/askerlee/craft.

show abstract

Section: Related Workmentioning

confidence: 99%

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

Sui¹,

Li²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Geometric priors are orthogonal to our approach and combining different forms of inductive biases is a promising direction for future work. Motion segmentation is concerned with separating objects from the background using optical flow [30,56,58]. Early approaches [7,35,43,43] tracked individual pixels with the flow and then clustered the resulting trajectories inspired by the common fate principle [38].…”

Section: Related Workmentioning

confidence: 99%

“…We experiment with two motion segmentation algorithms -a heuristic-based [35], and a learning-based one [15], for which we only use the motion stream trained on the toy Fliy-ingThings3D dataset [41]. Both methods take optical flow as input, so we evaluate them with both ground truth flow, and flow estimated with the state-of-the-art supervised [58] and unsupervised [56] approaches. Since the outputs of both methods contain many noisy segments, we apply a few generic post-processing steps to clean up the results.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…Qualitatively, this model, shown in the 4th column in Figure 4, has a slightly lower recall than the variant trained with ground truth moving segments due to the sparser learning signal. Intriguingly, replacing ground truth flow with the one estimated with a state-of-the-art supervised RAFT [58], or self-supervised SMURF [56] algorithms barely changes the performance, Learning-based TRI-PD KITTI SlotAttention [40] 10.2 13.8 MONet [8] 11.0 14.9 SCALOR [31] 18.6 21.1 S-IODINE [24] 9.8 14.4 MCG [3] 25.1 40.9 Ours 50.9 47.1…”

Section: Object Discovery In Realistic Videosmentioning

confidence: 99%

See 1 more Smart Citation

Discovering Objects that Can Move

Bao¹,

Tokmakov²,

Jabri³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper studies the problem of object discovery -separating objects from the background without manual labels. Existing approaches utilize appearance cues, such as color, texture, and location, to group pixels into object-like regions. However, by relying on appearance alone, these methods fail to separate objects from the background in cluttered scenes. This is a fundamental limitation since the definition of an object is inherently ambiguous and context-dependent. To resolve this ambiguity, we choose to focus on dynamic objects -entities that can move independently in the world. We then scale the recent auto-encoder based frameworks for unsupervised object discovery from toy synthetic images to complex real-world scenes. To this end, we simplify their architecture, and augment the resulting model with a weak learning signal from general motion segmentation algorithms. Our experiments demonstrate that, despite only capturing a small subset of the objects that move, this signal is enough to generalize to segment both moving and static instances of dynamic objects. We show that our model scales to a newly collected, photo-realistic synthetic dataset with street driving scenarios. Additionally, we leverage ground truth segmentation and flow annotations in this dataset for thorough ablation and evaluation. Finally, our experiments on the real-world KITTI benchmark demonstrate that the proposed approach outperforms both heuristic-and learning-based methods by capitalizing on motion cues.

show abstract

“…Self-supervised learning for optical flow. Significant progress has been achieved with selfsupervised learning for optical flow [20,24,30,38,56], focusing more on the loss than model architecture. UFlow [20] systematically studied a set of key components for self-supervised optical flow, including both model elements and training techniques.…”

Section: Previous Workmentioning

confidence: 99%

Disentangling Architecture and Training for Optical Flow

Sun¹,

Herrmann²,

Reda³

et al. 2022

Preprint

View full text Add to dashboard Cite

How important are training details and datasets to recent optical flow models like RAFT? And do they generalize? To explore these questions, rather than develop a new model, we revisit three prominent models, PWC-Net, IRR-PWC and RAFT, with a common set of modern training techniques and datasets, and observe significant performance gains, demonstrating the importance and generality of these training details. Our newly trained PWC-Net and IRR-PWC models show surprisingly large improvements, up to 30% versus original published results on Sintel and KITTI 2015 benchmarks. They outperform the more recent Flow1D on KITTI 2015 while being 3× faster during inference. Our newly trained RAFT achieves an Fl-all score of 4.31% on KITTI 2015, more accurate than all published optical flow methods at the time of writing. Our results demonstrate the benefits of separating the contributions of models, training techniques and datasets when analyzing performance gains of optical flow methods. Our source code will be publicly available.

show abstract

SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping

Cited by 55 publications

References 37 publications

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

Discovering Objects that Can Move

Disentangling Architecture and Training for Optical Flow

Contact Info

Product

Resources

About