Rendering Natural Camera Bokeh Effect with Deep Learning

Ignatov, Andrey; Patel, Jagruti; Timofte, Radu

doi:10.1109/cvprw50498.2020.00217

Cited by 77 publications

(69 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, further study is required to fully grasp and emulate the flexibility of the current mobile ISP pipelines. We refer the reader to [21] for an application of PyNET to rendering natural camera bokeh effect and employing a new "Everything is Better with Bokeh!" dataset of paired wide and shallow depth-of-field images.…”

Section: Discussionmentioning

confidence: 99%

Replacing Mobile Camera ISP with a Single Deep Learning Model

Ignatov

Gool

Timofte

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

158

127

View full text Add to dashboard Cite

As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP solutions. In this work, we demonstrate that even the most sophisticated ISP pipelines can be replaced with a single end-to-end deep learning model trained without any prior knowledge about the sensor and optics used in a particular device. For this, we present PyNET, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera, making the solution independent of any particular mobile ISP implementation. To validate the proposed approach on the real data, we collected a large-scale dataset consisting of 10 thousand full-resolution RAW-RGB image pairs captured in the wild with the Huawei P20 cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. The experiments demonstrate that the proposed solution can easily get to the level of the embedded P20's ISP pipeline that, unlike our approach, is combining the data from two (RGB + B/W) camera sensors. The dataset, pretrained models and codes used in this paper are available on the project website: https://people.ee.ethz. ch/˜ihnatova/pynet.html

show abstract

Section: Discussionmentioning

confidence: 99%

Replacing Mobile Camera ISP with a Single Deep Learning Model

Ignatov

Gool

Timofte

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

158

127

View full text Add to dashboard Cite

show abstract

“…One can see: (i) RVR produces images with strong artifacts around the boundaries; (ii) SteReFo synthesizes reasonable shallow DoF images, but it tends to blur the in-focus objects; (iii) The generalization of deep learning methods is limited. For example, PyNet sometimes focuses on the background by blurring the foreground objects as it is trained on EBB [ 50 ] with no knowledge of the focal plane on NJU2K; Instead, DL can generate plausible shallow DoF images due to the use of our SOD module. (iv) The transition of boundaries is too sharp in Ours (w/o depth) ; (v) Our predictions, by contrast, are clearer at the refocused plane and are more accurate around the boundaries.…”

Section: Methodsmentioning

confidence: 99%

“…To further verify the effectiveness of the overall framework, we compare with other state-of-the-art methods on the EBB [ 50 ] dataset. This dataset provides 4694 shallow/wide DoF image pairs captured by a Canon 7D DSLR with 50 mm f/1.8 lenses.…”

Section: Methodsmentioning

confidence: 99%

“…Therefore, some other methods [ 48 , 49 ] use off-the-shelf MDP methods to predict scene depth. In addition to explicitly using depth maps to render DOF images, some deep learning-based methods [ 18 , 50 , 51 ] propose to implicitly learn depth from all-in-focus and shallow DoF image pairs. Specifically, given an all-in-focus image as input, the network is optimized to render a synthetic shallow DOF image.…”

Section: Related Workmentioning

confidence: 99%

“…The monoculuar depth predition datasets can be found here: (HRWSI [ 14 ] at ), (NYUDv2 [ 64 ] at ), (Ibims [ 65 ] at ), (TUM [ 66 ] at ), (KITTI [ 67 ] at ), (Sintel [ 68 ] at ), and (DIODE [ 69 ] at ). The DoF related datasets can be found here: (4DLF [ 70 ] at ), (NJU2K [ 72 ] at ), and (EBB [ 50 ] at ), all accessed on 5 March 2021.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Ranking-Based Salient Object Detection and Depth Prediction for Shallow Depth-of-Field

Xian

Peng

Zhang

et al. 2021

Sensors

View full text Add to dashboard Cite

Shallow depth-of-field (DoF), focusing on the region of interest by blurring out the rest of the image, is challenging in computer vision and computational photography. It can be achieved either by adjusting the parameters (e.g., aperture and focal length) of a single-lens reflex camera or computational techniques. In this paper, we investigate the latter one, i.e., explore a computational method to render shallow DoF. The previous methods either rely on portrait segmentation or stereo sensing, which can only be applied to portrait photos and require stereo inputs. To address these issues, we study the problem of rendering shallow DoF from an arbitrary image. In particular, we propose a method that consists of a salient object detection (SOD) module, a monocular depth prediction (MDP) module, and a DoF rendering module. The SOD module determines the focal plane, while the MDP module controls the blur degree. Specifically, we introduce a label-guided ranking loss for both salient object detection and depth prediction. For salient object detection, the label-guided ranking loss comprises two terms: (i) heterogeneous ranking loss that encourages the sampled salient pixels to be different from background pixels; (ii) homogeneous ranking loss penalizes the inconsistency of salient pixels or background pixels. For depth prediction, the label-guided ranking loss mainly relies on multilevel structural information, i.e., from low-level edge maps to high-level object instance masks. In addition, we introduce a SOD and depth-aware blur rendering method to generate shallow DoF images. Comprehensive experiments demonstrate the effectiveness of our proposed method.

show abstract