No abstract

No abstract

Conventional approaches to image de-fencing have limited themselves to using only image data in adjacent frames of the captured video of an approximately static scene. In this work, we present a method to harness disparity using a stereo pair of fenced images in order to detect fence pixels. Tourists and amateur photographers commonly carry smartphones/phablets which can be used to capture a short video sequence of the fenced scene. We model the formation of the occluded frames in the captured video. Furthermore, we propose an optimization framework to estimate the de-fenced image using the total variation prior to regularize the ill-posed problem.

Tourists and amateur photographers are often hindered in capturing their cherished images/videos by a fence/occlusion that limits accessibility to the scene of interest. The situation has been exacerbated by growing concerns of security at public places and a need exists to provide a tool that can be used for post-processing such "fenced videos" to produce a "defenced" image. There are several challenges in this problem and in this work, we identify them as 1. Robust detection of the fence/occlusions. 2. Estimating pixel motion of background scene. 3. Filling in the fence/occlusions by utilizing information in multiple frames of the input video. We use a video captured by a camera panning the scene containing a fence and obtain a "de-fenced" image. Our method can effectively remove fences from images as demonstrated for several synthetic and real-world cases.Index Terms-Image de-fencing, inpainting, belief propagation, Markov random field. BACKGROUNDIn recent times, security concerns have led to extra precautions at popular public places and monuments such as fences and barricades. For the tourist, who wishes to capture his memories in an image/video at his favourite landmark, this poses a hindrance which spoils the captured data. It would be so much nice if a post-processing tool existed that can efficiently rid the input video of occlusion artifacts. It is common for the user to pan the camera while capturing a video of the scene in order to cover the entire landscape. A sample frame from a captured video is shown in Fig. 1 (a) wherein the fence is occluding parts of the face and body. We observe that the motion cue in video can be exploited to perform "de-fencing" of the degraded frames to obtain an image wherein the fence has been removed. In Fig. 1 (c), we show a sample output of the proposed algorithm which has successfully removed the occlusions due to fence pixels.There has been considerable progress in the area of image inpainting [3,4,5,6,7] in which most works assume that theFig. 1. Image de-fencing: (a) A frame from the video captured by panning the person occluded by a fence. (b) Estimating the global relative motion of background pixels by matching corresponding points using affine SIFT descriptor [1]. (c) De-fenced image obtained by the proposed algorithm. (d) A result from [2]. (e) Corresponding output of our technique.

Conventiona approaches to image de-fencing use multiple adjacent frames for segmentation of fences in the reference image and are limited to restoring images of static scenes only. In this paper, we propose a de-fencing algorithm for images of dynamic scenes using an occlusionaware optical flow method. We divide the problem of image de-fencing into the tasks of automated fence segmentation from a single image, motion estimation under known occlusions and fusion of data from multiple frames of a captured video of the scene. Specifically, we use a pre-trained convolutional neural network to segment fence pixels from a single image. The knowledge of spatial locations of fences is used to subsequently estimate optical flow in the occluded frames of the video for the final data fusion step. We cast the fence removal problem in an optimization framework by modeling the formation of the degraded observations. The inverse problem is solved using fast iterative shrinkage thresholding algorithm (FISTA). Experimental results show the effectiveness of proposed algorithm.