Deep Texture and Structure Aware Filtering Network for Image Smoothing

Abstract

Image smoothing is a fundamental task in computer vision, that aims to retain salient structures and remove insignificant textures. In this paper, we aim to address the fundamental shortcomings of existing image smoothing methods, which cannot properly distinguish textures and structures with similar low-level appearance. While deep learning approaches have started to explore the preservation of structure through image smoothing, existing work does not yet properly address textures. To this end, we generate a large dataset by blending natural textures with clean structure-only images, and then build a texture prediction network (TPN) that predicts the location and magnitude of textures. We then combine the TPN with a semantic structure prediction network (SPN) so that the final texture and structure aware filtering network (TSAFN) is able to identify the textures to remove ("texture-awareness") and the structures to preserve ("structure-awareness"). The proposed model is easy to understand and implement, and shows excellent performance on real images in the wild as well as our generated dataset.

PNEN: Pyramid Non-Local Enhanced Networks

“…Reference [29] presents an unsupervised deep model, which is optimized with a handcrafted objective function. Reference [30] attempts to identify out structures and textures in input images, facilitating structure and texture aware filtering. A CNN based pipeline is proposed for joint image filtering (e.g., RGB and depth images) in [1].…”

Edge‐aware image filtering using a structure‐guided CNN

“…This method trains the network using the gradient map obtained from the input image to reconstruct output images using the input image and the estimated feature map. Lu et al [19] proposed a three sub‐network model including structure prediction, texture prediction, and double‐GF layers. The smoothing result is obtained by the double‐guided layer that uses the prediction map and input image.…”

“…This method solves various problems such as smoothing [7] and dehazing using a deep convolutional network using adaptive batch normalisation and context aggregation network. The joint image filter model proposed by Lu et al [19] is based on the residual CNN to preserve structures of the target image using the guidance image. In the proposed method, feature maps are extracted from the target and guide layers by employing low‐depth layers with the skip connections.…”

Hybrid Loss for Learning Single-Image-based HDR Reconstruction

“…For example, convolutional GANs wellperformed in superresolution [21], denoising [3], or inpaiting [30]. Other than GANs, there are some promising approaches such as multiscale [38,24], perceptual losses [15], attention [31], or reinforcement learning [40,7]. While their insights are useful also for our task, such methods for LDR images are not directly applicable to HDR images.…”