Single Image Depth Estimation Trained via Depth From Defocus Cues

Abstract

Estimating depth from a single RGB images is a fundamental task in computer vision, which is most directly solved using supervised deep learning. In the field of unsupervised learning of depth from a single RGB image, depth is not given explicitly. Existing work in the field receives either a stereo pair, a monocular video, or multiple views, and, using losses that are based on structure-from-motion, trains a depth estimation network. In this work, we rely, instead of different views, on depth from focus cues. Learning is based on a novel Point Spread Function convolutional layer, which applies location specific kernels that arise from the Circle-Of-Confusion in each image location. We evaluate our method on data derived from five common datasets for depth estimation and lightfield images, and present results that are on par with supervised methods on KITTI and Make3D datasets and outperform unsupervised learning approaches. Since the phenomenon of depth from defocus is not dataset specific, we hypothesize that learning based on it would overfit less to the specific content in each dataset. Our experiments show that this is indeed the case, and an estimator learned on one dataset using our method provides better results on other datasets, than the directly supervised methods.Our method relies on a novel Point Spread Function (PSF) layer, which preforms a local operation over an image, with a location dependent kernel which is computed "on-the-fly", according to the estimated parameters of the PSF at each location. More specifically, the layer receives three inputs: an all-in-focus image, estimated depth-map and camera parameters, and outputs an image at one specific focus. This image is then compared to the training images to compute a loss. Both the forward and backward operations of the layer are efficiently computed using a dedicated CUDA kernel. This layer is then used as part of a novel architecture, combining the successful ASPP architecture [5,9]. To improve the ASPP block, we add dense connections [16], followed by self-attention [42].We evaluate our method on all relevant benchmarks we were able to obtain. These include the flower lightfield dataset and the multifocus indoor and outdoor scene dataset, for which we compare the ability to generate unseen focus arXiv:2001.05036v1 [cs.CV]

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

“…Zhang et al [24] considered the training loss as combination of the point-to-point, shape and distribution similarity between predictions and ground truth, which leveraged hierarchical structure information to guide the network optimization. Alhashim et al [37] and Gur et al [38] regarded the training loss as the sum of L1 loss and structural similarity (SSIM) loss, which sought the balance between the point-to-point difference and the distortions of high frequency details in the image domain. Hu et al [25] and Chen et al [27] made a simple analysis of orthogonal sensitivities to different types of errors and then proposed to use a combination of three loss function terms, which included the point-to-point loss, the gradients loss, and the normal loss.…”

Learning Depth for Scene Reconstruction Using an Encoder-Decoder Model

“…We investigate classic models, including different layers of CNNs, such as ResNet-18, ResNet-34, ResNet-50, DenseNet-121, DPN-68, DPN-92, and DPN-131. In these CNNs, ResNet-50 is employed as an encoder by famous studies [6], [8], [32], [33]. Although few methods use DPNs to predict depth, we modify DPN-92 as the encoder of DEM after considering the tradeoff between multiply-andaccumulate operations and accuracy of DPNs.…”

“…In dense depth prediction, our decoder is superior because the pixel-level depth maps are learned by transposed convolution and convolution layers. By contrast, the state of the arts [5], [8], [32] use decoders including linear interpolation.…”

“…Different models are evaluated on the NYU-Depth-v2 testing dataset. In Table 4 and Table 5, the results of models [1], [5]- [7], [8], [15], [24], [25], [28], [32], [36], [39], [43] are taken from their respective papers. The results of models [16], [33], [37] are acquired by implementing their respective methods.…”

“…The results of models [16], [33], [37] are acquired by implementing their respective methods. In the work [32], we select the first training setup because it consumes less calculation than other training setups. In the research [33], we adopt the original parameter setup to obtain better results than other parameter setups.…”