Deep Recurrent Neural Network with Multi-Scale Bi-directional Propagation for Video Deblurring

Abstract

The success of the state-of-the-art video deblurring methods stems mainly from implicit or explicit estimation of alignment among the adjacent frames for latent video restoration. However, due to the influence of the blur effect, estimating the alignment information from the blurry adjacent frames is not a trivial task. Inaccurate estimations will interfere the following frame restoration. Instead of estimating alignment information, we propose a simple and effective deep Recurrent Neural Network with Multi-scale Bi-directional Propagation (RNN-MBP) to effectively propagate and gather the information from unaligned neighboring frames for better video deblurring. Specifically, we build a Multi-scale Bi-directional Propagation (MBP) module with two U-Net RNN cells which can directly exploit the inter-frame information from unaligned neighboring hidden states by integrating them in different scales. Moreover, to better evaluate the proposed algorithm and existing state-of-the-art methods on real-world blurry scenes, we also create a Real-World Blurry Video Dataset (RBVD) by a well-designed Digital Video Acquisition System (DVAS) and use it as the training and evaluation dataset. Extensive experimental results demonstrate that the proposed RBVD dataset effectively improve the performance of existing algorithms on real-world blurry videos, and the proposed algorithm performs favorably against the state-of-the-art methods on three typical benchmarks. The code is available at https://github.com/XJTU-CVLAB-LOWLEVEL/RNN-MBP.

UNet Based Multi-Scale Recurrent Network for Lightweight Video Deblurring

“…Su et al [10] TSP [22] RNN-MBP [2] MPRNet [9] NAFNet [18] BasicVSR++ [5] VRT [ Method EDVR [8] Su et al [10] STFAN [23] TSP [22] RNN-MBP [2] FGST [24] BasicVSR++ [5] VRT […”

“…Second, networks that require a lot of computational complexity have recently been proposed for video deblurring. Specifically, there are VRT [1], which is a network using transformers, and RNN-MBP [2], which uses a lot of high-resolution processing. However, video deblurring is most likely to be used in situations where computational cost is limited.…”

Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events

“…The spatio-temporal correlation between adjacent inputs is critical for video deblurring. Recurrent neural network (RNN) or convolutional neural network (CNN) are adopted to exploit temporal information (Nah, Son, and Lee 2019;Zhong et al 2020;Zhou et al 2019;Su et al 2017;Zhu et al 2022). To improve the deblurring performance further, some extra multiple frames aligning methods were proposed to model spatio-temporal correlation, such as optical flow based methods (Pan, Bai, and Tang 2020;Xiang, Wei, and Pan 2020), deformable and dynamic convolutions based methods (Wang et al 2019;Zhou et al 2019).…”

“…However, it is challenging to design such priors and constraints to model the inherent properties of latent frames and motion blur. Due to the success of deep neural networks (DNNs), some deep convolutional neural network (CNN)-based methods (Zhang et al 2019;Zamir et al 2021;Chen et al 2021;Cho et al 2021), recurrent neural network (RNN)-based methods (Nah, Son, and Lee 2019;Zhong et al 2020;Zhou et al 2019;Zhu et al 2022) and Transformer-based methods (Liang et al 2021;Wang et al 2022b;Liang et al 2022b,a) have been proposed for motion deblurring, which implicitly learn more general prior information from large-scale training data. Despite their good performance, these learning-based deblurring methods may fail to deal with severe blur.…”

Video restoration based on deep learning: a comprehensive survey

“…Such unidirectional propagation may result to be suboptimal because the amount of information received when processing different frames is different, as the first frames have access to less information than the last ones. Some methods (Huang et al 2017b;Chan et al 2021aChan et al , 2022Zhu et al 2022) use bidirectional information propagation, where information is propagated both forward and backward so that each frame can also benefit from the information coming from subsequent frames. Chan et al (2021a) conducted a study demonstrating that bidirectional propagation improves the restoration performance.…”