2HDED:Net for Joint Depth Estimation and Image Deblurring from a Single Out-of-Focus Image

Abstract

Depth estimation and all-in-focus image restoration from defocused RGB images are related problems, although most of the existing methods address them separately. The few approaches that solve both problems use a pipeline processing to derive a depth or defocus map as an intermediary product that serves as a support for image deblurring, which remains the primary goal. In this paper, we propose a new Deep Neural Network (DNN) architecture that performs in parallel the tasks of depth estimation and image deblurring, by attaching them the same importance. Our Two-headed Depth Estimation and Deblurring Network (2HDED:NET) is an encoderdecoder network for Depth from Defocus (DFD) that is extended with a deblurring branch, sharing the same encoder. The network is tested on NYU-Depth V2 dataset and compared with several state-of-the-art methods for depth estimation and image deblurring.

iDFD: A Dataset Annotated for Depth and Defocus

“…In this section, we evaluate iDFD dataset by training a multi-task network that is challenging for both predicting depth and deblurring RGB images. The network is called 2HDED:NET [12]. To emphasize on the importance of using real data, we retrain the network in the same conditions on NYU dataset, completed with synthetically defocused images.…”

“…Finally, on iDFD, we test the network trained on NYU to evaluate the performance of a model learned from similar but synthetic data. 2HDED:NET is a recently proposed architecture consisting of one encoder and two decoders [12]. The encoder is fed in with a defocused RGB image, while one decoder outputs the scene depth map and the other deblurred image.…”

“…Afterwords, the network is fine tuned on a small proprietary dataset containing naturally defocused images captured with a DSLR camera. A similar situation is encountered in [12], where the multitask network 2HDED-NET is trained with synthetically defocused images and depth GT from NYU v2 and Make3D datasets, in order to estimate both the depth map and AiF image. These examples show how necessary are the datasets with naturally defocused, AiF images and depth GT.…”

“…Despite this strong relationship, image deblurring and DFD were treated separately by the overwhelming majority of the DL based methods. Only recently, the authors of [12] proposed a multi-task network called 2HDED:NET, where DFD and deblurring support each other to better accomplish these tasks. But again, 2HDED:NET is trained on a synthetically defocused dataset without confirming its effectiveness on naturally defocused images because of the lack of such dataset [13].…”

Learning-Based Proof of the State-of-the-Art Geometric Hypothesis on Depth-of-Field Scaling and Shifting Influence on Image Sharpness

“…Their proposed model performed well with both sharp and blurred images in computational depth estimation up to a range of 3.3 m, regardless of whether the image was in focus or out of focus. Nazir et al [20] suggested a deep convolutional neural network to estimate the depth and image deblurring. Kumar et al [21] presented a novel technique to generate a more accurate depth map for dynamic scenes using a combination of defocus and motion cues.…”