Fast and High-Quality Image Denoising via Malleable Convolutions

Abstract

Many image processing networks apply a single set of static convolutional kernels across the entire input image, which is sub-optimal for natural images, as they often consist of heterogeneous visual patterns. Recent works in classification, segmentation, and image restoration have demonstrated that dynamic kernels outperform static kernels at modeling local image statistics. However, these works often adopt per-pixel convolution kernels, which introduce high memory and computation costs. To achieve spatial-varying processing without significant overhead, we present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. The weights of Mal-leConv are dynamically produced by an efficient predictor network capable of generating content-dependent outputs at specific spatial locations. Unlike previous works, Malle-Conv generates a much smaller set of spatially-varying kernels from input, which enlarges the network's receptive field and significantly reduces computational and memory costs. These kernels are then applied to a full-resolution feature map through an efficient slice-and-conv operator with minimum memory overhead. We further build an efficient denoising network using MalleConv, coined as MalleNet. It achieves high quality results without very deep architecture, e.g., reaching 8.91× faster speed compared to the best performed denoising algorithms (SwinIR), while maintaining similar performance. We also show that a single Malle-Conv added to a standard convolution-based backbone can contribute significantly to reducing the computational cost or boosting image quality at a similar cost. Project page: https://yifanjiang.net/MalleConv.html

Adaptive Dynamic Filtering Network for Image Denoising

“…These persistent performance improvements also demonstrate that our ADFNet has powerful capabilities to restore strong corrupted images. Besides, some CNN-and Transformer-based approaches (Liang et al 2021;Jiang et al 2022) 2021) takes more than ×8 time and ×4 FLOPs to process one 128 × 128 image compared to our model. However, the PSNR value on Kodak24 is still lower than ours.…”