On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation

Abstract

Scene depth estimation from stereo and monocular imagery is critical for extracting 3D information for downstream tasks such as scene understanding. Recently, learning-based methods for depth estimation have received much attention due to their high performance and flexibility in hardware choice. However, collecting ground truth data for supervised training of these algorithms is costly or outright impossible. This circumstance suggests a need for alternative learning approaches that do not require corresponding depth measurements. Indeed, self-supervised learning of depth estimation provides an increasingly popular alternative. It is based on the idea that observed frames can be synthesized from neighboring frames if accurate depth of the scene is known -or in this case, estimated. We show empirically that -contrary to common belief -improvements in image synthesis do not necessitate improvement in depth estimation. Rather, optimizing for image synthesis can result in diverging performance with respect to the main prediction objective -depth. We attribute this diverging phenomenon to aleatoric uncertainties, which originate from data. Based on our experiments on four datasets (spanning street, indoor, and medical) and five architectures (monocular and stereo), we conclude that this diverging phenomenon is independent of the dataset domain and not mitigated by commonly used regularization techniques. To underscore the importance of this finding, we include a survey of methods which use image synthesis, totaling 127 papers over the last six years. This observed divergence has not been previously reported or studied in depth, suggesting room for future improvement of self-supervised approaches which might be impacted the finding.

On the Uncertain Single-View Depths in Endoscopies

“…Self-supervised depth learning uses image synthesis to compute the photometric loss. However, traditional synthesis losses to regress depth are limited due to the aleatoric uncertainty of input images [50]. Poggi et al [12] are the first ones addressing such problem by including aleatoric uncertainty in the self-supervised monocular loss.…”

A Disparity Refinement Framework for Learning-based Stereo Matching Methods in Cross-domain Setting for Laparoscopic Images

“…Learning-based stereo matching methods can be roughly divided into fullysupervised methods [15,16] and self-supervised methods [17,18]. The former uses accurate ground truth disparity maps for training, while the latter relies on formulating image synthesis loss.…”

Disparity refinement framework for learning-based stereo matching methods in cross-domain setting for laparoscopic images

“…13,14 Learning-based stereo matching methods could be classified into fully-supervised methods 15,16 and self-supervised methods. 17,18 The former class uses accurate ground truth disparity maps for training, whereas the latter class relies on formulating image synthesis loss. In the era of deep learning, learning-based stereo matching methods are reported to achieve high performance on several public benchmark datasets and outperform traditional methods.…”