Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

Abstract

Deep learning approaches to optical flow estimation have seen rapid progress over the recent years. One common trait of many networks is that they refine an initial flow estimate either through multiple stages or across the levels of a coarse-to-fine representation. While leading to more accurate results, the downside of this is an increased number of parameters. Taking inspiration from both classical energy minimization approaches as well as residual networks, we propose an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks. It reduces the number of parameters, improves the accuracy, or even achieves both. Moreover, we show that integrating occlusion prediction and bi-directional flow estimation into our IRR scheme can further boost the accuracy. Our full network achieves stateof-the-art results for both optical flow and occlusion estimation across several standard datasets. Number of parameters (million) AEPE on Sintel Train Clean FlowNetC FlowNetS SpyNetOurs (PWC-Net + Occ) PWC-NetOurs (PWC-Net + Bi) Ours (IRR-PWC) LiteFlowNetOurs (PWC-Net + IRR)

Large Displacement Optical Flow Estimation Based on Robust Interpolation of Sparse Correspondences

“…To make a convincing comparison, we compare the flow result of our method with that of several state-of-the-art methods including Classic+NL [22], Deepflow [50], LDOF [47], JOF [26], STDC-Flow [54], PWC-Net [36], FlowNet2.0 [31] and IRR-PWC [39], in which the Classic+NL and JOF methods belonging to the variational optical flow approach, the Deepflow, LDOF and STDC-Flow methods belonging to the matching-based optical flow approach, and the PWC-Net, FlowNet2.0 and IRR-PWC methods are CNN-based optical flow approach.…”

“…In this paper, we apply a 7-level pyramid to the PWC-Net model and set a search range of 4 pixels to compute the cost volume at each level. We train the PWC-Net model on FlyingChairs datasets by using a long learning rate and use a batch size Finally, the IRR-PWC method [39] incorporates an iterative residual refinement scheme into the PWC-Net framework, which improves the robustness of optical flow in regions of occlusions. In this paper, we use the same network parameters and training strategy with that of PWC-Net method to test the performance of IRR-PWC.…”

Implicit and Explicit Regularization for Optical Flow Estimation

“…The authors claimed that progressively increasing the difficulty of training samples plays a critical role on accuracy; therefore, they proposed a stacked, progressively trained architecture with intermediate image warping operations, setting a new SoA. Hur et al [ 26 ] claimed that stacked architectures benefit from reusing the same module for iterative residual refinement, instead of stacking independent modules. Hui et al [ 27 ] tried to reduce the computational requirements of Ilg’s method.…”

AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

“…The BOE IOT AIBD IMP team has proposed the frame interpolation solution based on DAIN [2]. The team has improved the model by using IRR-PWC [15] for calculating the optical flows, PacJointUpsample [46] for joint optical flow upsampling, VNL [53] for predicting the depth maps and pixel-adaptive convolutions [46] in frame synthesis module for generating the final results. Based on CyclicGen [24], a new two-stage cyclic generation process is proposed for training the 4x interpolation model as in Figure 11.…”

“…Based on CyclicGen [24], a new two-stage cyclic generation process is proposed for training the 4x interpolation model as in Figure 11. The method is using pre-trained IRR-PWC [15], VNL [53], PacJointUpsample [46], and some sub-module/layers of DAIN [2], and was trained/finetuned on the REDS VTSR dataset. The entire network architecture is visualized in Figure 12.…”