My camera can see through fences: A deep learning approach for image de-fencing

Abstract

In recent times, the availability of inexpensive image capturing devices such as smartphones/tablets has led to an exponential increase in the number of images/videos captured. However, sometimes the amateur photographer is hindered by fences in the scene which have to be removed after the image has been captured. Conventional approaches to image de-fencing suffer from inaccurate and non-robust fence detection apart from being limited to processing images of only static occluded scenes. In this paper, we propose a semi-automated de-fencing algorithm using a video of the dynamic scene. We use convolutional neural networks for detecting fence pixels. We provide qualitative as well as quantitative comparison results with existing lattice detection algorithms on the existing PSU NRT data set [1] and a proposed challenging fenced image dataset. The inverse problem of fence removal is solved using split Bregman technique assuming total variation of the de-fenced image as the regularization constraint.

De-Fencing and Multi-Focus Fusion Using Markov Random Field and Image Inpainting

“…Restoration homographic matrices, dictionary based on k-means labeling, exemplar based inpainting manual initialization Deep learning based methods [11], [12], [13], [14], [15], [16], [17], [18], [19], [20] De-fencing adversarial, structural [21], [22], [23], [24], [25] Fusion multi-scale decomposition, dictionary-learning, nuclear norm regularizer, morphologies constraints, adaptive fusion rules, fractional differential coefficients, geometric sparse coefficients overcomplete dictionary, patch based clustering, single dictionary learning time efficiency, separate fusion and noise removal tasks, information loss due to channel-wise processing Model based methods [26], [27], [28], [29], [30], [31], [32], [33] [34], [35], [36], [37] Fusion SSIM, encoder features, K-means clustering, NSCT, Coupled-Neural-Ps consistency verification photo realistic fusion, blocking artifacts, post processing complete contours; and generator-discriminator setting for prediction of contour completion. In [3], subtraction based on gray-scale binarization is applied on multi-focus auxiliary images to obtain initial mask for image inpainting.…”

“…The optical flow estimate (while blurring the fence regions) is used in split Bregman optimization method while considering total variation as a regularization constraint. In [15], a modified convolutional neural network (CNN) based deep learning technique uses multiple stereo frames for fence detection. This scheme shows good performance, however, sometimes it fails to restore fences due to inaccurate motion estimation.…”

Single-Image Fence Removal Using Deep Convolutional Neural Network

“…In static captured videos, hidden part at a certain frame will become visible in another frame. Methods [4], [5] tackle not only static scenes but also dynamic scenes. Recently, some deep learning based methods have been proposed.…”

Stereo image de-fencing using smartphones

“…Yi et al [8] proposed a bottom-up framework for fence detection by clustering pixels into coherent groups using color and motion features through graph-cut based optimization. In our previous works [9,16] we have proposed a supervised learning approach for automatic identification of occlusions/fences using only image data.…”