Synthetic depth-of-field with a single-camera mobile phone

Abstract

a) Input image with detected face (d) Our output synthetic shallow depth-of-eld image (b) Person segmentation mask (c) Mask + disparity from DP Fig. 1. We present a system that uses a person segmentation mask (b) and a noisy depth map computed using the camera's dual-pixel (DP) auto-focus hardware (c) to produce a synthetic shallow depth-of-field image (d) with a depth-dependent blur on a mobile phone. Our system is marketed as "Portrait Mode" on several Google-branded phones. Shallow depth-of-field is commonly used by photographers to isolate a subject from a distracting background. However, standard cell phone cameras cannot produce such images optically, as their short focal lengths and small apertures capture nearly all-in-focus images. We present a system to computationally synthesize shallow depth-of-field images with a single mobile camera and a single button press. If the image is of a person, we use a person segmentation network to separate the person and their accessories from the background. If available, we also use dense dual-pixel auto-focus hardware, effectively a 2-sample light field with an approximately 1 millimeter baseline, to compute a dense depth map. These two signals are combined and used to render a defocused image. Our system can process a 5.4 megapixel image in 4 seconds on a mobile phone, is fully automatic, and is robust enough to be used by non-experts. The modular nature of our system allows it to degrade naturally in the absence of a dual-pixel sensor or a human subject.

Single image portrait relighting

“…Complete relighting: Figures 1 and 9 show the results of using our model for the task of complete relighting -replacing the illumination of the scene. The foreground masks produced by Wadhwa et al [2018] are used to isolate the portrait's subject before it is used as input to our model, and our renderings are produced by compositing the relit foreground over a synthetic rendering of the target illumination. Despite significant variation in the skin tones and ages of the subjects, and in the camera angle and initial illumination, our model is capable of producing realistically relit renderings.…”

“…The resolution of our OLAT images is 2400 × 1800. For each image, we first use the human segmentation algorithm [Wadhwa et al 2018] to mask out the background. Then we take a random crop from the OLAT imageset, with a position chosen uniformly at random and with the crop size chosen uniformly from [512, 1024] at random.…”

“…9. Examples of our model being used for complete relighting on "in the wild" portrait images taken from the Portrait-mode and HDR+ dataset [Wadhwa et al 2018;Hasinoff et al 2016]. Using only an RGB input image (column 1) and the portrait mask produced by [Wadhwa et al 2018], we are capable of producing relit images according to arbitrary environment illuminations (columns 2 and 3) for a wide variety of subjects and viewing angles.…”

Learning Single Camera Depth Estimation Using Dual-Pixels

“…1, the quality of depth maps that can be produced by conventional stereo techniques is limited, because the interplay between disparity and focus in DP imagery can cause classic stereo-matching techniques to fail. Existing monocular learning-based techniques also perform poorly on this (a) RGB image (b) Depth from [55] (c) Our depth Here we have an RGB image (a) containing dual-pixel data. Crops of the left and right dual-pixel images corresponding to the marked rectangle in (a) are shown in (d), (e), and their intensity profiles along the marked scanline are shown in (f).…”

SteReFo: Efficient Image Refocusing with Stereo Vision

“…capture, such as portrait images. To address this problem a trend has emerged, where shallow depth of field images are computationally synthesized from all-in-focus images [50], usually by leveraging a depth estimation. In the rest of the paper we refer to this task as refocusing.…”