Learning Brightness Transfer Functions for the Joint Recovery of Illumination Changes and Optical Flow

Abstract

Abstract. The increasing importance of outdoor applications such as driver assistance systems or video surveillance tasks has recently triggered the development of optical flow methods that aim at performing robustly under uncontrolled illumination. Most of these methods are based on patch-based features such as the normalized cross correlation, the census transform or the rank transform. They achieve their robustness by locally discarding both absolute brightness and contrast. In this paper, we follow an alternative strategy: Instead of discarding potentially important image information, we propose a novel variational model that jointly estimates both illumination changes and optical flow. The key idea is to parametrize the illumination changes in terms of basis functions that are learned from training data. While such basis functions allow for a meaningful representation of illumination effects, they also help to distinguish real illumination changes from motion-induced brightness variations if supplemented by additional smoothness constraints. Experiments on the KITTI benchmark show the clear benefits of our approach. They do not only demonstrate that it is possible to obtain meaningful basis functions, they also show state-of-the-art results for robust optical flow estimation.

Optical flow modeling and computation: A survey

“…The offset formulation is also used in [10,88], but is associated to a sparsity constraint aiming at retrieving violations due to occlusions. The more general approach of [74] parametrizes intensity changes in terms of Brightness Transfer Function [106], which coefficients are learned from training data. The model (15) is based on a general polynomial approximation.…”

On robust optical flow estimation on image sequences with differently exposed frames using primal–dual optimization

“…Image sequences from the KITTI benchmark suite [39] are known to typically include natural illumination changes between the captured frames. As a result, the estimation performance is improved when accounting for illumination changes in the OF cost functional modeling [27,40,20]. Due to the nature of the KITTI data, consisting largely of at surfaces, we exchange the spatial TV regularization in E S by the second order Total Generalized Variation [25] that penalizes deviations from (piecewise) ane ow solutions.…”

Direct Variational Perspective Shape from Shading with Cartesian Depth Parametrisation

“…where Ψ is the Charbonnier function [13] Ψ (s 2 ) = 2λ 2 1 + s 2 λ 2 (15) with contrast parameter λ . Such higher-order smoothness terms have already been successfully applied in the context of perspective SfS parametrised in terms of the radial depth [29], orthographic SfS [49], image denoising [31], optical lithography [21] and motion estimation [18]. Finally, the use of the confidence function c in the data term allows to exclude unreliable image regions which have been identified a priori, e.g.…”