SteReFo: Efficient Image Refocusing with Stereo Vision

Abstract

Whether to attract viewer attention to a particular object, give the impression of depth or simply reproduce humanlike scene perception, shallow depth of field images are used extensively by professional and amateur photographers alike. To this end, high quality optical systems are used in DSLR cameras to focus on a specific depth plane while producing visually pleasing bokeh. We propose a physically motivated pipeline to mimic this effect from all-in-focus stereo images, typically retrieved by mobile cameras. It is capable to change the focal plane a posteriori at 76 FPS on KITTI [13] images to enable realtime applications. As our portmanteau suggests, SteReFo interrelates stereo-based depth estimation and refocusing efficiently. In contrast to other approaches, our pipeline is simultaneously fully differentiable, physically motivated, and agnostic to scene content. It also enables computational video focus tracking for moving objects in addition to refocusing of static images. We evaluate our approach on publicly available datasets [13,33,9] and quantify the quality of architectural changes.

Ranking-Based Salient Object Detection and Depth Prediction for Shallow Depth-of-Field

“…Thus, some methods use RGB-D images [ 45 , 46 , 47 ] and stereo image pairs [ 3 , 4 ] to render DoF images. For example, SteReFo [ 3 ] interrelates stereo-based depth estimation and refocusing effectively. However, such methods rely on specific hardware, e.g., RGB-D sensors and stereo cameras.…”

“…Xu et al [ 1 ] learn a spatially-variant RNN [ 52 ] filter to render a shallow DoF image from a portrait photo. Besides, some approaches [ 2 , 3 , 5 ] which manually select a focal plane, have also been proposed for DoF rendering. By contrast, this paper proposes a method to automatically render a shallow DoF image from an arbitrary natural image.…”

“…However, these acquired photos are always all-in-focus due to the narrow baseline and fixed aperture. Hence, more and more attention has been paid to the shallow DoF rendering techniques [ 1 , 2 , 3 , 4 , 5 ] in recent years.…”

“…To render realistic shallow DoF, depth information is required. Some methods use stereo techniques to compute depth maps from stereo images [ 3 , 4 ] and dual-pixel data [ 2 ]. However, such methods depend on specific hardware (e.g., stereo cameras or dual-pixel sensors) to capture two views.…”

“…Traditional rendering methods always use the gather and scatter operators to render shallow DoF. However, these methods [ 2 , 3 , 18 ] in practice utilize the layered depth rendering strategy that applies a blur kernel to each depth plane. To keep the refocused plane clear and enable smooth transition around depth discontinuities, our method processes each pixel one by one.…”