Adversarial Spatio-Temporal Learning for Video Deblurring

Abstract

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and the temporal domain (i.e., neighboring frames) and 2) how to restore sharp image details with respect to the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a deblurring network (DBLRNet) for spatial-temporal learning by applying a 3D convolution to both the spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to the improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the generative adversarial network (GAN) architecture and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as deblurring GAN, is tested on two standard benchmarks and achieves the state-of-the-art performance.

Deep Video Deblurring: The Devil is in the Details

“…Our new fan max initialization combined with linear outputs already yields a substantial 2dB benefit over our sigmoid baseline. (2) While recent work proposes to deblur in YCbCr color space [57], we show that there is no significant benefit over RGB. Instead, a simple extension of the training schedule can lead to an additional 0.4dB benefit.…”

“…We do not rely on a recurrent architecture, but a plain CNN, achieving very competitive results. Zhang et al [57] use spatio-temporal 3D convolutions in the early stages of a deep residual network. Chen et al [3] extend [26] with a physics-based reblurring pipeline, which constructs a reblurred image from the sharp predictions using optical flow, and subsequently enforces consistency between the reblurred image and the blurry input image.…”

“…While this is clearly not necessary in deep neural architectures -we can just output three color channels simultaneously -the question remains whether the RGB color space is appropriate. Zhang et al [57] propose to convert the blurry input images to YCbCr space, where Y corresponds to grayscale intensities and CbCr denotes the color components, c.f . Fig.…”

“…Patch size and sequence length. Much of previous work [3,22,46,57] is trained on random crops of size 128 2 , yet the significance of this choice is not further justified. In general, larger crops are beneficial as they reduce the influence of boundaries, given the typically big receptive fields.…”

Cousin Network Guided Sketch Recognition via Latent Attribute Warehouse

“…Recent years have witnessed the success of deep learning in various tasks of computer vision, such as object detection (Ren et al 2015), image recognition Szegedy et al 2015;, object tracking , facial recognition (Sun, Wang, and Tang 2013;Zhang et al 2015; and multimedia analysis (Simonyan and Zisserman 2014;Ledig et al 2017;Xiong et al 2018;Zhang et al 2019). For sketch recognition, a variant of Siamese CNN is proposed in (Wang, Kang, and Li 2015) to match sketch and photo without special process of sketch images.…”