Optical Flow Estimation Using a Spatial Pyramid Network

Abstract

We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning. This estimates large motions in a coarse-to-fine approach by warping one image of a pair at each pyramid level by the current flow estimate and computing an update to the flow. Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flow update. Unlike the recent FlowNet approach, the networks do not need to deal with large motions; these are dealt with by the pyramid. This has several advantages. First, our Spatial Pyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in terms of model parameters. This makes it more efficient and appropriate for embedded applications. Second, since the flow at each pyramid level is small (< 1 pixel), a convolutional approach applied to pairs of warped images is appropriate. Third, unlike FlowNet, the learned convolution filters appear similar to classical spatio-temporal filters, giving insight into the method and how to improve it. Our results are more accurate than FlowNet on most standard benchmarks, suggesting a new direction of combining classical flow methods with deep learning.

CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency

“…Results. We compare our approach with the state-of-theart methods [43,34,21]. Table 4 shows that our method achieves improved performance on both datasets.…”

Deep HDR Video from Sequences with Alternating Exposures

“…Analysis of the Results Figure 4: On the left, we compare our flow network against the FlowNet [IMS * 17] and SPyNet [RB17] by producing an HDR frame from the POKER FULLSHOT scene. Note that, we trained both the FlowNet and SPyNet networks in combination with our merge network (Sec.…”

“…For the flow network, we build upon the hierarchical coarse-to-fine architecture, concurrently proposed by Ranjan and Black[RB17] andWang et al [WZK * 17], and incorporate the three c 2019 The Author(s) Computer Graphics Forum c 2019 The Eurographics Association and John Wiley & Sons Ltd.…”

Adaptive Motion Pooling and Diffusion for Optical Flow Computation

“…By considering state of the art computer vision approaches [16], our model (average EPE for all the sequences, aEPE=0.71 pixel) performs better than some algorithms, e.g. FlowNetC (aEPE=0.93 pixel), but other algorithms outperform it, e.g.…”

“…Very few attempts have been made to incorporate these ideas into spatio-temporal filter based models, and given the recent growth in neuroscience, it is very interesting to revisit this model incorporating the new findings and examining the efficacy. Differently from FFV1MT and Spynet [16], which only rely on scale space for diffusion of non-local cues, our AMPD model provides a clue on the potential role played by the recurrent interactions in solving the blank wall problem by non local cue propagation. It is also worth noting that bilateral filtering based techniques are gaining popularity in semantic segmentation using convolutional neural networks.…”