Deep Learning Based Fence Segmentation and Removal from an Image Using a Video Sequence

Abstract

Conventiona approaches to image de-fencing use multiple adjacent frames for segmentation of fences in the reference image and are limited to restoring images of static scenes only. In this paper, we propose a de-fencing algorithm for images of dynamic scenes using an occlusionaware optical flow method. We divide the problem of image de-fencing into the tasks of automated fence segmentation from a single image, motion estimation under known occlusions and fusion of data from multiple frames of a captured video of the scene. Specifically, we use a pre-trained convolutional neural network to segment fence pixels from a single image. The knowledge of spatial locations of fences is used to subsequently estimate optical flow in the occluded frames of the video for the final data fusion step. We cast the fence removal problem in an optimization framework by modeling the formation of the degraded observations. The inverse problem is solved using fast iterative shrinkage thresholding algorithm (FISTA). Experimental results show the effectiveness of proposed algorithm.

Accurate and Efficient Video De-Fencing Using Convolutional Neural Networks and Temporal Information

“…To recover the occluded background by a fence, we use the occlusion-aware optical flow [6] which is improved from the coarse-to-fine optical flow framework [17]. Let Y o and Y k refer to the target image and the k th neighboring image, respectively.…”

“…This process consists of selecting meaningful frames and determining the content at each pixel. For selecting frames, [6] manually selected the frames for simplicity. However, it is not feasible for video de-fencing.…”

“…Comparison of de-fencing results. Images in the first row and in the second row are the results by [6] and by the proposed algorithm, respectively. The left and right images in the third row show the zoom-in patches of the first row and the second row, respectively.…”

“…This method requires scribbles along the fence from a user, limiting its efficiency. To avoid scribbles from humans, Jonna et al proposed a deep learning-based method for fence segmentation [4][5][6]. The main idea is detecting the joints of fences using convolutional neural networks (CNN) and support vector machine (SVM) and then connecting the joints to obtain scribbles for image matting.…”

“…It also does not require any manual labeling and is able to handle irregular fences and challenging scenes such as having similar color for both background and fences. Moreover, we propose a fast and robust recovery method using occlusion-aware optical flow [6]. The recovery method produces more plausible de-fenced images and videos.…”