Imposing Consistency for Optical Flow Estimation

Abstract

We propose a novel data augmentation approach, Dis-tractFlow, for training optical flow estimation models by introducing realistic distractions to the input frames. Based on a mixing ratio, we combine one of the frames in the pair with a distractor image depicting a similar domain, which allows for inducing visual perturbations congruent with natural objects and scenes. We refer to such pairs as distracted pairs. Our intuition is that using semantically meaningful distractors enables the model to learn related variations and attain robustness against challenging deviations, compared to conventional augmentation schemes focusing only on low-level aspects and modifications. More specifically, in addition to the supervised loss computed between the estimated flow for the original pair and its ground-truth flow, we include a second supervised loss defined between the distracted pair's flow and the original pair's ground-truth flow, weighted with the same mixing ratio. Furthermore, when unlabeled data is available, we extend our augmentation approach to self-supervised settings through pseudo-labeling and cross-consistency regularization. Given an original pair and its distracted version, we enforce the estimated flow on the distracted pair to agree with the flow of the original pair. Our approach allows increasing the number of available training pairs significantly without requiring additional annotations. It is agnostic to the model architecture and can be applied to training any optical flow estimation models. Our extensive evaluations on multiple benchmarks, including Sintel, KITTI, and SlowFlow, show that DistractFlow improves existing models consistently, outperforming the latest state of the art.

SC-AOF: A Sliding Camera and Asymmetric Optical Flow-Based Blending Method for Image Stitching

“…The general pipeline of the optical flow calculation is to construct an image pyramid, calculate the optical flow of each layer from coarse to fine, and use the estimated current-layer optical flow divided by the scaling factor as the initial optical flow of the finer layer until the optical flow of the finest layer is obtained [23][24][25]. Different methods are proposed to achieve better solutions that satisfy brightness constancy assumptions, solve large displacements and appearance variation [27,28], address edge blur and improve temporal consistency [29][30][31]. Recently, some deep learning methods are proposed.…”

InertialNet: Inertial Measurement Learning for Simultaneous Localization and Mapping

“…To improve the direct-regression optical flow estimation, an effective global matching step was introduced before optimization. In [ 38 ], consistency learning strategies were proposed for optical flow estimation. By utilizing the consistencies on occlusion and transformation, the technique is able to learn the description of pixel-level motion without additional annotation.…”

Relation Extraction on the Wargame based on Pre-trained Models

“…Lai et al [26] propose to learn a discriminator on warp errors from ground truth warps to provide feedback on the overall quality of the warps,replacing the noisy per-pixel color cues. Jeong et al [27] add a separate segmentation module to a flow network, supervise this module with occlusion masks applied onto unlabelled images, and also ask flows to be consistent across transformations applied to the input, which is similar to our goal but requires modified architecture for segmentation.…”