Disentangling Architecture and Training for Optical Flow

Sun, Deqing; Herrmann, Charles; Reda, Fitsum A.; Rubinstein, Michael; Fleet, David J.; Freeman, William T.

doi:10.1007/978-3-031-20047-2_10

Cited by 17 publications

(11 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LiteFlowNet2 [HTL20] is already an optimized version of FlowNet 2.0 [IMS * 17], in comparison PWC‐Net [SYLK18] has more potential for optimization/compression. Moreover, recently it has been shown that PWC‐Net can achieve similar accuracy to RAFT when trained on a large‐scale synthetic dataset [SVH * 21] and that PWC‐Net achieves favorable trade‐offs vs. other state‐of‐the‐art methods when selecting for runtime performance or higher image resolutions [SHR * 22]. Hence, we select PWC‐Net for further compression.…”

Section: Methodsmentioning

confidence: 99%

Interactive Control over Temporal Consistency while Stylizing Video Streams

Shekhar

Reimann

Hilscher

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Image stylization has seen significant advancement and widespread interest over the years, leading to the development of a multitude of techniques. Extending these stylization techniques, such as Neural Style Transfer (NST), to videos is often achieved by applying them on a per‐frame basis. However, per‐frame stylization usually lacks temporal consistency, expressed by undesirable flickering artifacts. Most of the existing approaches for enforcing temporal consistency suffer from one or more of the following drawbacks: They (1) are only suitable for a limited range of techniques, (2) do not support online processing as they require the complete video as input, (3) cannot provide consistency for the task of stylization, or (4) do not provide interactive consistency control. Domain‐agnostic techniques for temporal consistency aim to eradicate flickering completely but typically disregard aesthetic aspects. For stylization tasks, however, consistency control is an essential requirement as a certain amount of flickering adds to the artistic look and feel. Moreover, making this control interactive is paramount from a usability perspective. To achieve the above requirements, we propose an approach that stylizes video streams in real‐time at full HD resolutions while providing interactive consistency control. We develop a lite optical‐flow network that operates at 80 Frames per second (FPS) on desktop systems with sufficient accuracy. Further, we employ an adaptive combination of local and global consistency features and enable interactive selection between them. Objective and subjective evaluations demonstrate that our method is superior to state‐of‐the‐art video consistency approaches. maxreimann.github.io/stream‐consistency

show abstract

Section: Methodsmentioning

confidence: 99%

Interactive Control over Temporal Consistency while Stylizing Video Streams

Shekhar

Reimann

Hilscher

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Optical Flow Estimation: Several deep architectures have been proposed for optical flow [4,8,23,29,30,38]. Among these, Recurrent All Pairs Field Transforms (RAFT) [30] have shown significant performance improvement over previous methods, inspiring many subsequent works [6,14,26,27,35]. Following the structure of RAFT architecture, complementary studies [12,14,33,35,39] proposed advancements on feature extraction, 4D correlation volume, recurrent update blocks, and more recently, transformer extensions [6,39].…”

Section: Related Workmentioning

confidence: 99%

“…Though its performance gain is notable, Aut-oFlow employs synthetic augmentations. The work in [27] utilizes AutoFlow and argues that it is important to disentangle architecture and training pipeline. [27] also points out that some of the performance improvements of the recent methods are due to hyperparameters, dataset extensions, and training optimizations.…”

Section: Related Workmentioning

confidence: 99%

“…The work in [27] utilizes AutoFlow and argues that it is important to disentangle architecture and training pipeline. [27] also points out that some of the performance improvements of the recent methods are due to hyperparameters, dataset extensions, and training optimizations. Our work focuses on more capable augmentations for model agnostic training, not architecture novelties.…”

Section: Related Workmentioning

confidence: 99%

“…Among the latest works, many focus on developing novel neural network architectures, such as PWC-Net [29], RAFT [30], and FlowFormer [6]. Other stud- † Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. ies investigate how to improve different aspects of supervised training [27], e.g., gradient clipping, learning rate, and training compute load. More related to our paper are those incorporating data augmentation during training (e.g., [30]), including color jittering, random occlusion, cropping, and flipping.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Imposing Consistency for Optical Flow Estimation

Jeong

Lin

Porikli

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We propose a novel data augmentation approach, Dis-tractFlow, for training optical flow estimation models by introducing realistic distractions to the input frames. Based on a mixing ratio, we combine one of the frames in the pair with a distractor image depicting a similar domain, which allows for inducing visual perturbations congruent with natural objects and scenes. We refer to such pairs as distracted pairs. Our intuition is that using semantically meaningful distractors enables the model to learn related variations and attain robustness against challenging deviations, compared to conventional augmentation schemes focusing only on low-level aspects and modifications. More specifically, in addition to the supervised loss computed between the estimated flow for the original pair and its ground-truth flow, we include a second supervised loss defined between the distracted pair's flow and the original pair's ground-truth flow, weighted with the same mixing ratio. Furthermore, when unlabeled data is available, we extend our augmentation approach to self-supervised settings through pseudo-labeling and cross-consistency regularization. Given an original pair and its distracted version, we enforce the estimated flow on the distracted pair to agree with the flow of the original pair. Our approach allows increasing the number of available training pairs significantly without requiring additional annotations. It is agnostic to the model architecture and can be applied to training any optical flow estimation models. Our extensive evaluations on multiple benchmarks, including Sintel, KITTI, and SlowFlow, show that DistractFlow improves existing models consistently, outperforming the latest state of the art.

show abstract

MS-RAFT+: High Resolution Multi-Scale RAFT

Jahedi,

Luz,

Rivinius

et al. 2023

Int J Comput Vis

View full text Add to dashboard Cite

Hierarchical concepts have proven useful in many classical and learning-based optical flow methods regarding both accuracy and robustness. In this paper we show that such concepts are still useful in the context of recent neural networks that follow RAFT’s paradigm refraining from hierarchical strategies by relying on recurrent updates based on a single-scale all-pairs transform. To this end, we introduce MS-RAFT+: a novel recurrent multi-scale architecture based on RAFT that unifies several successful hierarchical concepts. It employs a coarse-to-fine estimation to enable the use of finer resolutions by useful initializations from coarser scales. Moreover, it relies on RAFT’s correlation pyramid that allows to consider non-local cost information during the matching process. Furthermore, it makes use of advanced multi-scale features that incorporate high-level information from coarser scales. And finally, our method is trained subject to a sample-wise robust multi-scale multi-iteration loss that closely supervises each iteration on each scale, while allowing to discard particularly difficult samples. In combination with an appropriate mixed-dataset training strategy, our method performs favorably. It not only yields highly accurate results on the four major benchmarks (KITTI 2015, MPI Sintel, Middlebury and VIPER), it also allows to achieve these results with a single model and a single parameter setting. Our trained model and code are available at https://github.com/cv-stuttgart/MS_RAFT_plus.

show abstract

Disentangling Architecture and Training for Optical Flow

Cited by 17 publications

References 53 publications

Interactive Control over Temporal Consistency while Stylizing Video Streams

Interactive Control over Temporal Consistency while Stylizing Video Streams

Imposing Consistency for Optical Flow Estimation

MS-RAFT+: High Resolution Multi-Scale RAFT

Contact Info

Product

Resources

About