Supervised Raw Video Denoising With a Benchmark Dataset on Dynamic Scenes

Abstract

In recent years, raw video denoising has garnered increased attention due to the consistency with the imaging process and well-studied noise modeling in the raw domain. Despite these advancements, two problems still hinder the denoising performance. Firstly, there is no large dataset with realistic motions for supervised raw video denoising, as capturing noisy and clean frames for real dynamic scenes is difficult. To address this, we propose recapturing existing high-resolution videos displayed on a 4K screen. Specifically, we recapture the screen content with high-low ISO settings to construct noisy-clean paired frames. Afterward, we introduce intensity, spatial, and color correction strategies to make the paired frames well-aligned. Then, the aligned frames are concatenated with temporal order to construct paired videos. In this way, we construct a video denoising dataset (named as ReCRVD) with 120 groups of noisy-clean videos, whose ISO values ranging from 1600 to 25600. Secondly, while non-local temporal-spatial attention is beneficial for denoising, it often leads to heavy computation costs. In this work, we propose an efficient raw video denoising transformer network (RViDeformer) that explores both short and long-distance correlations. Specifically, we introduce Low-Resolution-Window Self-Attention (LWSA), Global-Window Self-Attention (GWSA), and Neighbour-Window Self-Attention (NWSA) to build a multi-branch spatial self-attention for spatial reconstruction. Similarly, Global-Window Temporal Mutual Attention (GTMA) and Neighbour-Window Temporal Mutual Attention (NTMA) are proposed to build multi-branch temporal self-attention for temporal reconstruction. We employ reparameterization to reduce computation costs. Our network is trained in both supervised and unsupervised manners, achieving the best performance compared with state-of-the-art methods. Additionaly, the model trained with our proposed dataset (ReCRVD) outperforms the model trained with previous benchmark dataset (CRVD) when evaluated on the real-world outdoor noisy videos. Our code and dataset will be released after the acceptance of this work.

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

“…In 2017, Plötz and Roth [38] established a real raw image denoising benchmark, which showed that these deep denoisers failed to generalize beyond the synthetic data used during training and were outperformed by standard non-learned methods, such as BM3D [11]. Subsequent work on both single [5,9] and multi-image [8,18,36,49] denoising demonstrated the benefits of training networks to operate directly on noisy raw input data. Modern cellphone camera pipelines perform a robust averaging of multiple noisy input frames in the raw domain [20], though they typically cannot afford to employ deep networks due to speed and power limitations.…”

“…Recent years have seen an increasing focus on developing deep learning methods for denoising images directly in the raw linear domain [5,9]. This effort has expanded to include multi-image denoisers that can be applied to burst images or video frames [8,43,49]. These multi-image denoisers typically assume that there is a relatively small amount of motion between frames, but that there may be large amounts of object motion within the scene.…”

An Innovative Approach for Denoising of the Digital Video Stream

“…al. [7] have proposed an technique for video denoising using supervised learning approach. Authors have attempted to rectify the critical issues by designing motions for controllabel objects.…”

Concurrent Video Denoising and Deblurring for Dynamic Scenes

“…The problem of spatially variant motion blur, related to independently moving objects in the presence of noise, has not yet been addressed in the deep learning literature. Not only is it an intrinsically challenging problem, but relevant research is also limited by the difficulty in constructing such labeled datasets [17], [18]. For instance, [12] used a beam splitter to construct real blurry-sharp frames, whereas [18] emulated motion by manually moving objects and used a fixed tripod to capture multiple frames of the same scene before generating real noisy-clean pairs of frames by averaging the noisy instantiations of the same scene.…”

“…Not only is it an intrinsically challenging problem, but relevant research is also limited by the difficulty in constructing such labeled datasets [17], [18]. For instance, [12] used a beam splitter to construct real blurry-sharp frames, whereas [18] emulated motion by manually moving objects and used a fixed tripod to capture multiple frames of the same scene before generating real noisy-clean pairs of frames by averaging the noisy instantiations of the same scene. In this study, we rely on the realistic blurry dataset of [12] and the realistic Poisson-Gaussian noise model [18], [19].…”