End-to-End Training of Hybrid CNN-CRF Models for Stereo

Abstract

We propose a novel and principled hybrid CNN+CRF model for stereo estimation. Our model allows to exploit the advantages of both, convolutional neural networks (CNNs) and conditional random fields (CRFs) in an unified approach. The CNNs compute expressive features for matching and distinctive color edges, which in turn are used to compute the unary and binary costs of the CRF. For inference, we apply a recently proposed highly parallel dual block descent algorithm which only needs a small fixed number of iterations to compute a high-quality approximate minimizer. As the main contribution of the paper, we propose a theoretically sound method based on the structured output support vector machine (SSVM) to train the hybrid CNN+CRF model on large-scale data end-to-end. Our trained models perform very well despite the fact that we are using shallow CNNs and do not apply any kind of postprocessing to the final output of the CRF. We evaluate our combined models on challenging stereo benchmarks such as Middlebury 2014 and Kitti 2015 and also investigate the performance of each individual component.

Parametric Majorization for Data-Driven Energy Minimization Methods

“…Several techniques have been developed in the field of structured support vector machines (SSVMs) [92,28,1,95] that are very relevant to the task of learning energy models, as SSVMs can be understood as bi-level problems with a lower-level energy that is linear in θ and often a noncontinuous higher-level loss. Various strategies such as margin rescaling [92], slack rescaling [95,97], softmaxmargins [40] exist and have also been applied recently in the training of computer vision models in [54,29], we will later return to their connection to the investigated strategies.…”

Learned Collaborative Stereo Refinement

“…(ii) This positive trend is transferred to the test set for the average error and the RMS error. (iii) The bad{0.5, 1} errors on the test set are reduced and (iv) the bad{2, 4} errors slightly increase on the test set compared to [11]. One reason for this is the limited amount of training data for these very high-resolution images.…”

“…We use our method on top of the CNN-CRF [11] stereo method for the official test set evaluation (see Table 2). We set the temperature parameter η = 0.075 in all experiments.…”

SDNet: Semantically Guided Depth Estimation Network

“…Supervised learning techniques include models which are trained on stereo images, but which can infer depth maps on monocular images. [12] propose an approach which follows this paradigm. They use the correlations of CNN feature maps of stereo images and derive the unary matching costs.…”