Scalable Full Flow with Learned Binary Descriptors

Abstract

Abstract. We propose a method for large displacement optical flow in which local matching costs are learned by a convolutional neural network (CNN) and a smoothness prior is imposed by a conditional random field (CRF). We tackle the computation-and memory-intensive operations on the 4D cost volume by a min-projection which reduces memory complexity from quadratic to linear and binary descriptors for efficient matching. This enables evaluation of the cost on the fly and allows to perform learning and CRF inference on high resolution images without ever storing the 4D cost volume. To address the problem of learning binary descriptors we propose a new hybrid learning scheme. In contrast to current state of the art approaches for learning binary CNNs we can compute the exact non-zero gradient within our model. We compare several methods for training binary descriptors and show results on public available benchmarks.

Learning Energy Based Inpainting for Optical Flow

“…cost volumes are constructed. We employ the method of [23] and generate 2 3D volumes from one complete 4D cost volume (c.f. Sec.…”

“…For feature generation we follow [23] and employ a Siamese network consisting of two convolutional branches with shared parameters. In our implementation we utilize a feed forward network comprised of 5 convolutional layers with a filter size of 3 × 3 and 64 channels, followed by a single 1-D convolution.…”

“…(3). To that end, [23] proposes to use min-projection to eliminate one of the motion directions via: cor j i (x, y, u i ) := min u1−i∈H cor j (x, y, u 0 , u 1 ). Most prominently this reduces the memory complexity of storing the cost volume from quadratic to linear.…”

“…Compared to nearest neighbor field methods [18,36], we make use of a complete cost-volume and avoid a coarse-to-fine scheme or hashing. To that end, we make use of a recent result [23] that allows for a low memory footprint of the cost volume. In contrast to network based solutions to inpainting [41], we maintain the interpretability of an energy based framework.…”

Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems

“…which follows the scalable model of Munda et al [34], avoiding the storage of all matching scores that for an M ×N image have the size M ×N ×D 2 . The inner maximization steps correspond to the first iteration of an approximate MAP inference [34].…”