Seeing through the fence: Image de-fencing using a video sequence

Abstract

Tourists and amateur photographers are often hindered in capturing their cherished images/videos by a fence/occlusion that limits accessibility to the scene of interest. The situation has been exacerbated by growing concerns of security at public places and a need exists to provide a tool that can be used for post-processing such "fenced videos" to produce a "defenced" image. There are several challenges in this problem and in this work, we identify them as 1. Robust detection of the fence/occlusions. 2. Estimating pixel motion of background scene. 3. Filling in the fence/occlusions by utilizing information in multiple frames of the input video. We use a video captured by a camera panning the scene containing a fence and obtain a "de-fenced" image. Our method can effectively remove fences from images as demonstrated for several synthetic and real-world cases.Index Terms-Image de-fencing, inpainting, belief propagation, Markov random field. BACKGROUNDIn recent times, security concerns have led to extra precautions at popular public places and monuments such as fences and barricades. For the tourist, who wishes to capture his memories in an image/video at his favourite landmark, this poses a hindrance which spoils the captured data. It would be so much nice if a post-processing tool existed that can efficiently rid the input video of occlusion artifacts. It is common for the user to pan the camera while capturing a video of the scene in order to cover the entire landscape. A sample frame from a captured video is shown in Fig. 1 (a) wherein the fence is occluding parts of the face and body. We observe that the motion cue in video can be exploited to perform "de-fencing" of the degraded frames to obtain an image wherein the fence has been removed. In Fig. 1 (c), we show a sample output of the proposed algorithm which has successfully removed the occlusions due to fence pixels.There has been considerable progress in the area of image inpainting [3,4,5,6,7] in which most works assume that theFig. 1. Image de-fencing: (a) A frame from the video captured by panning the person occluded by a fence. (b) Estimating the global relative motion of background pixels by matching corresponding points using affine SIFT descriptor [1]. (c) De-fenced image obtained by the proposed algorithm. (d) A result from [2]. (e) Corresponding output of our technique.

A multimodal approach for image de-fencing and depth inpainting

“…We compare our method with the approach proposed in [1]. The technique in [1] makes the assumption that the frames of the captured video are relatively shifted by global pixel motion.…”

“…Hence, it cannot handle the scenario wherein there are multiple depths in the scene to be de-fenced. We give as input to [1] the motion of the background pixels in the 3 frames as the global motion and the output is shown in Fig. 4 (f).…”

“…The technique in [1] makes the assumption that the frames of the captured video are relatively shifted by global pixel motion. Hence, it cannot handle the scenario wherein there are multiple depths in the scene to be de-fenced.…”

“…This is a non-trivial problem and has been addressed in the past [1], [2], [3]. However, unlike our multimodal method the works of [1], [2], [3] address this task only with image data and do not have access to depth maps of the occluded scene. The major novelty of our work is that we have captured aligned depth data along with a color video of the scene using the Kinect sensor.…”

“…Our multi-modal approach is related to prior work on image de-fencing using only image data [1], [2], [3]. The problem of image de-fencing and depth completion can be divided into three challenging tasks.…”