Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation

Zhang, Feihu; Woodford, Oliver J.; Prisacariu, Victor Adrian; Torr, Philip H. S.

doi:10.1109/iccv48922.2021.01063

Cited by 77 publications

(48 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The vast majority of neural network-based optical flow method considers the task of estimating dense pixel displacements from a pair of frames [41,44,40,16,22,43,9,2,49,18,50,51,8]. A common component of many highly successful methods are explicit correlation volumes that guide the matching process.…”

Section: Image-basedmentioning

confidence: 99%

“…A common component of many highly successful methods are explicit correlation volumes that guide the matching process. This inductive bias enables high performance and data efficiency [41] as well as strong cross-dataset generalization [49]. Multi-Frame optical flow estimation has been mostly explored in the selfsupervised learning setting [15,19,28] or optimization-based literature [20,37,11].…”

Section: Image-basedmentioning

confidence: 99%

See 1 more Smart Citation

Dense Continuous-Time Optical Flow from Events and Frames

Gehrig¹,

Muglikar²,

Scaramuzza³

2022

Preprint

View full text Add to dashboard Cite

We present a method for estimating dense continuous-time optical flow. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute perpixel, continuous-time optical flow by additionally using events from an event camera. Events provide temporally fine-grained information about movement in image space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous-time via parameterized Bézier curves. To achieve this, we introduce multiple innovations to build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. The proposed approach outperforms existing image-based and event-based methods by 11.5 % lower EPE on DSEC-Flow. Finally, we introduce a novel synthetic dataset MultiFlow for pixel trajectory regression on which our method is currently the only successful approach.

show abstract

Section: Image-basedmentioning

confidence: 99%

Section: Image-basedmentioning

confidence: 99%

Dense Continuous-Time Optical Flow from Events and Frames

Gehrig¹,

Muglikar²,

Scaramuzza³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Within this paradigm, some works improve the efficiency of the correlation volume, such as VCN [45] and DICL [41]. Similar to our objective, Separable Flow [47] aims to improve the accuracy of the correlation volume, by decomposing the 4D correlation volume into two 3D volumes, for the uand v-directional flow regression, respectively. Separable Flow essentially imposes stronger inductive biases to obtain more accurate correlations than RAFT, as well as more accurate flow 1 .…”

Section: Related Workmentioning

confidence: 99%

“…• Separable Flow [47] decomposes the 4D correlation volume as two 3D volumes for the u and v directions.…”

Section: Standard Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

Sui¹,

Li²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Optical flow estimation aims to find the 2D motion field by identifying corresponding pixels between two images. Despite the tremendous progress of deep learning-based optical flow methods, it remains a challenge to accurately estimate large displacements with motion blur. This is mainly because the correlation volume, the basis of pixel matching, is computed as the dot product of the convolutional features of the two images. The locality of convolutional features makes the computed correlations susceptible to various noises. On large displacements with motion blur, noisy correlations could cause severe errors in the estimated flow. To overcome this challenge, we propose a new architecture "CRoss-Attentional Flow Transformer" (CRAFT), aiming to revitalize the correlation volume computation. In CRAFT, a Semantic Smoothing Transformer layer transforms the features of one frame, making them more global and semantically stable. In addition, the dot-product correlations are replaced with transformer Cross-Frame Attention. This layer filters out feature noises through the Query and Key projections, and computes more accurate correlations. On Sintel (Final) and KITTI (foreground) benchmarks, CRAFT has achieved new state-of-the-art performance. Moreover, to test the robustness of different models on large motions, we designed an image shifting attack that shifts input images to generate large artificial motions. Under this attack, CRAFT performs much more robustly than two representative methods, RAFT and GMA. The code of CRAFT is is available at https://github.com/askerlee/craft.

show abstract

FlowFormer: A Transformer Architecture for Optical Flow

Huang

Shi

Zhang

et al. 2022

Lecture Notes in Computer Science

112

View full text Add to dashboard Cite

This paper introduces a novel transformer-based network architecture, FlowFormer, along with the Masked Cost Volume AutoEncoding (MCVA) for pretraining it to tackle the problem of optical flow estimation. FlowFormer tokenizes the 4D cost-volume built from the source-target image pair and iteratively refines flow estimation with a cost-volume encoder-decoder architecture. The cost-volume encoder derives a cost memory with alternate-group transformer (AGT) layers in a latent space and the decoder recurrently decodes flow from the cost memory with dynamic positional cost queries. On the Sintel benchmark, FlowFormer architecture achieves 1.16 and 2.09 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the GMA (1.388 and 2.47). MCVA enhances FlowFormer by pretraining the cost-volume encoder with a masked autoencoding scheme, which further unleashes the capability of FlowFormer with unlabeled data. This is especially critical in optical flow estimation because ground truth flows are more expensive to acquire than labels in other vision tasks. MCVA improves FlowFormer all-sided and FlowFormer+MCVA ranks 1st among all published methods on both Sintel and KITTI-2015 benchmarks and achieves the best generalization performance. Specifically, FlowFormer+MCVA achieves 1.07 and 1.94 AEPE on the Sintel benchmark, leading to 7.76% and 7.18% error reductions from FlowFormer.

show abstract

Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation

Cited by 77 publications

References 65 publications

Dense Continuous-Time Optical Flow from Events and Frames

Dense Continuous-Time Optical Flow from Events and Frames

CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow

FlowFormer: A Transformer Architecture for Optical Flow

Contact Info

Product

Resources

About