OIFlow: Occlusion-Inpainting Optical Flow Estimation by Unsupervised Learning

Liu, Shuaicheng; Luo, Kun; Ye, Nianjin; Wang, Chuan; Wang, Jue; Zeng, Bing

doi:10.1109/tip.2021.3093781

Cited by 29 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DistillFlow [43] trains multiple teacher models and introduces a confidence based two-stage distillation approach for improvement. OIFlow [44] puts up an occlusion-inpainting framework to make full use of occlusion regions. Recently, ASFlow [45] presents content-aware pooling and adaptive flow upsampling modules to improve pyramidbased unsupervised flow deep structure.…”

Section: B Learning Unsupervised Optical Flowmentioning

confidence: 99%

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

Kong

Yang

2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.

show abstract

Section: B Learning Unsupervised Optical Flowmentioning

confidence: 99%

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

Kong

Yang

2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Several works [28], [60], [61], [62] focus on dealing with the occlusion problem by forwardbackward occlusion checking, range-map occlusion checking, data distillation, and augmentation regularization loss. Other methods concentrate on optical flow learning by improving image alignment, including the census loss [60], formulation of multiframes [63], epipolar constraints [64], depth constraints [65], feature similarity constraints [66], and occlusion inpainting [67]. UFlow [68] proposed a unified framework to systematically analyze and integrate different unsupervised components.…”

Section: Optical Flowmentioning

confidence: 99%

GyroFlow+: Gyroscope-Guided Unsupervised Deep Homography and Optical Flow Learning

Li¹,

Luo²,

Zeng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Existing homography and optical flow methods are erroneous in challenging scenes, such as fog, rain, night, and snow because the basic assumptions such as brightness and gradient constancy are broken. To address this issue, we present an unsupervised learning approach that fuses gyroscope into homography and optical flow learning. Specifically, we first convert gyroscope readings into motion fields named gyro field. Second, we design a self-guided fusion module (SGF) to fuse the background motion extracted from the gyro field with the optical flow and guide the network to focus on motion details. Meanwhile, we propose a homography decoder module (HD) to combine gyro field and intermediate results of SGF to produce the homography. To the best of our knowledge, this is the first deep learning framework that fuses gyroscope data and image content for both deep homography and optical flow learning. To validate our method, we propose a new dataset that covers regular and challenging scenes. Experiments show that our method outperforms the state-of-the-art methods in both regular and challenging scenes.

show abstract

“…Comparisons with unsupervised methods. We use the conventional unsupervised algorithm [81], [82], [83] (using photometic loss and smooth loss) to pretrain FlowFormer for comparison. Our MCVA outperforms the unsupervised counterpart.…”

Section: Ablation Study On Mcva Pretrainingmentioning

confidence: 99%

“…Comparisons with Unsupervised Methods. We also use conventional unsupervised methods to pretrain Flow-Former with photometric loss and smooth loss following [81], [82], [83] and then finetune it in the 'C+T' setting as MVCA-FlowFormer. As shown in Tab.…”

Section: Size Of Q Lmentioning

confidence: 99%

FlowFormer: A Transformer Architecture for Optical Flow

Huang

Shi

Zhang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper introduces a novel transformer-based network architecture, FlowFormer, along with the Masked Cost Volume AutoEncoding (MCVA) for pretraining it to tackle the problem of optical flow estimation. FlowFormer tokenizes the 4D cost-volume built from the source-target image pair and iteratively refines flow estimation with a cost-volume encoder-decoder architecture. The cost-volume encoder derives a cost memory with alternate-group transformer (AGT) layers in a latent space and the decoder recurrently decodes flow from the cost memory with dynamic positional cost queries. On the Sintel benchmark, FlowFormer architecture achieves 1.16 and 2.09 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the GMA (1.388 and 2.47). MCVA enhances FlowFormer by pretraining the cost-volume encoder with a masked autoencoding scheme, which further unleashes the capability of FlowFormer with unlabeled data. This is especially critical in optical flow estimation because ground truth flows are more expensive to acquire than labels in other vision tasks. MCVA improves FlowFormer all-sided and FlowFormer+MCVA ranks 1st among all published methods on both Sintel and KITTI-2015 benchmarks and achieves the best generalization performance. Specifically, FlowFormer+MCVA achieves 1.07 and 1.94 AEPE on the Sintel benchmark, leading to 7.76% and 7.18% error reductions from FlowFormer.

show abstract

OIFlow: Occlusion-Inpainting Optical Flow Estimation by Unsupervised Learning

Cited by 29 publications

References 40 publications

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

GyroFlow+: Gyroscope-Guided Unsupervised Deep Homography and Optical Flow Learning

FlowFormer: A Transformer Architecture for Optical Flow

Contact Info

Product

Resources

About