FlowFormer: A Transformer Architecture for Optical Flow

Abstract

This paper introduces a novel transformer-based network architecture, FlowFormer, along with the Masked Cost Volume AutoEncoding (MCVA) for pretraining it to tackle the problem of optical flow estimation. FlowFormer tokenizes the 4D cost-volume built from the source-target image pair and iteratively refines flow estimation with a cost-volume encoder-decoder architecture. The cost-volume encoder derives a cost memory with alternate-group transformer (AGT) layers in a latent space and the decoder recurrently decodes flow from the cost memory with dynamic positional cost queries. On the Sintel benchmark, FlowFormer architecture achieves 1.16 and 2.09 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the GMA (1.388 and 2.47). MCVA enhances FlowFormer by pretraining the cost-volume encoder with a masked autoencoding scheme, which further unleashes the capability of FlowFormer with unlabeled data. This is especially critical in optical flow estimation because ground truth flows are more expensive to acquire than labels in other vision tasks. MCVA improves FlowFormer all-sided and FlowFormer+MCVA ranks 1st among all published methods on both Sintel and KITTI-2015 benchmarks and achieves the best generalization performance. Specifically, FlowFormer+MCVA achieves 1.07 and 1.94 AEPE on the Sintel benchmark, leading to 7.76% and 7.18% error reductions from FlowFormer.

Imposing Consistency for Optical Flow Estimation

“…• We demonstrate the effectiveness of DistractFlow in supervised [6,14,30] and semi-supervised settings, showing that DistractFlow outperforms the very recent FlowSupervisor [9] that require additional in-domain unlabeled data.…”

“…Optical Flow Estimation: Several deep architectures have been proposed for optical flow [4,8,23,29,30,38]. Among these, Recurrent All Pairs Field Transforms (RAFT) [30] have shown significant performance improvement over previous methods, inspiring many subsequent works [6,14,26,27,35]. Following the structure of RAFT architecture, complementary studies [12,14,33,35,39] proposed advancements on feature extraction, 4D correlation volume, recurrent update blocks, and more recently, transformer extensions [6,39].…”

“…Among these, Recurrent All Pairs Field Transforms (RAFT) [30] have shown significant performance improvement over previous methods, inspiring many subsequent works [6,14,26,27,35]. Following the structure of RAFT architecture, complementary studies [12,14,33,35,39] proposed advancements on feature extraction, 4D correlation volume, recurrent update blocks, and more recently, transformer extensions [6,39]. In DistractFlow, we introduce a new modelagnostic training method that can help any model.…”

“…Recent years have seen significant progress in optical flow estimation thanks to the development of deep learning, e.g., [4,7,8,23]. Among the latest works, many focus on developing novel neural network architectures, such as PWC-Net [29], RAFT [30], and FlowFormer [6]. Other stud- † Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. ies investigate how to improve different aspects of supervised training [27], e.g., gradient clipping, learning rate, and training compute load.…”

Asymmetric Bilateral Motion Estimation for Video Frame Interpolation

“…Attempts have been made to develop effective VFI methods [1,2,6,[9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27]. Especially, with the advances in optical flow estimation [28][29][30][31][32][33][34][35][36][37], motion-based VFI methods provide remarkable performances. But, VFI for high-resolution videos, e.g.…”