GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences

Abstract

Establishing dense correspondences between a pair of images is an important and general problem, covering geometric matching, optical flow and semantic correspondences. While these applications share fundamental challenges, such as large displacements, pixel-accuracy, and appearance changes, they are currently addressed with specialized network architectures, designed for only one particular task. This severely limits the generalization capabilities of such networks to new scenarios, where e.g. robustness to larger displacements or higher accuracy is required.In this work, we propose a universal network architecture that is directly applicable to all the aforementioned dense correspondence problems. We achieve both high accuracy and robustness to large displacements by investigating the combined use of global and local correlation layers. We further propose an adaptive resolution strategy, allowing our network to operate on virtually any input image resolution. The proposed GLU-Net achieves stateof-the-art performance for geometric and semantic matching as well as optical flow, when using the same network and weights. Code and trained models are available at https://github.com/PruneTruong/GLU-Net.

Learning Accurate Dense Correspondences and When to Trust Them

“…Fundamentally, the goal of probabilistic deep learning is to achieve a predictive model p(y|X; θ) that coincides with [34] between the flow y estimated by GLU-Net [64] and the ground-truth y. empirical probabilities as well as possible. We can get important insights into this problem by studying the empirical error distribution of a state-of-the-art matching model, in this case GLU-Net [64], as shown in Fig. 3.…”

“…This forces the network to focus on the appearance of the image region in order to predict its motion and uncertainty. Given a base flow Ỹ relating Ĩr to Ĩq and representing a simple transformation such as a homography as in prior works [40,46,63,64], we create a residual flow = i ε i , by adding small local perturbations ε i . The query image I q = Ĩq is left unchanged while the reference I r is generated by warping Ĩr according to the residual flow .…”

“…We adopt the recent GLU-Net-GOCor [63,64] as our base architecture. It consists in a four-level pyramidal network operating at two image resolutions and employing a VGG-16 network [5] pre-trained on ImageNet for feature extraction.…”

“…This leads to large displacements and significant appearance transformations between the frames. In contrast to optical flow, the more general dense correspondence problem has received much less attention [40,48,53,64]. Dense flow estimation is prone to errors in the presence of large displacements, appearance changes, or homogeneous regions.…”

“…However, learning reliable and generalizable uncertainties without densely annotated realworld training data is a highly challenging problem. Standard self-supervised techniques [40,46,64] do not faithfully model real motion patterns, appearance changes, and occlusions. We tackle this challenge by introducing a carefully designed architecture and improved self-supervision to ensure robust and generalizable uncertainty predictions.…”