Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal Transforms

Abstract

Abstract-We propose a powerful video filtering algorithm that exploits temporal and spatial redundancy characterizing natural video sequences. The algorithm implements the paradigm of nonlocal grouping and collaborative filtering, where a higherdimensional transform-domain representation of the observations is leveraged to enforce sparsity and thus regularize the data: 3-D spatiotemporal volumes are constructed by tracking blocks along trajectories defined by the motion vectors. Mutually similar volumes are then grouped together by stacking them along an additional fourth dimension, thus producing a 4-D structure, termed group, where different types of data correlation exist along the different dimensions: local correlation along the two dimensions of the blocks, temporal correlation along the motion trajectories, and nonlocal spatial correlation (i.e. self-similarity) along the fourth dimension of the group. Collaborative filtering is then realized by transforming each group through a decorrelating 4-D separable transform and then by shrinkage and inverse transformation. In this way, the collaborative filtering provides estimates for each volume stacked in the group, which are then returned and adaptively aggregated to their original positions in the video. The proposed filtering procedure addresses several video processing applications, such as denoising, deblocking, and enhancement of both grayscale and color data. Experimental results prove the effectiveness of our method in terms of both subjective and objective visual quality, and shows that it outperforms the state of the art in video denoising.

Video Denoising via Empirical Bayesian Estimation of Space-Time Patches

“…Some sample sequences can be downloaded from the project website 7 in uncompressed format. In Table 2 we compare our results against the state-of-the-art method V-BM4D [39] with two different choices of parameters. The method labeled V-BM4D-tip corresponds to the results reported in [39].…”

“…In Table 2 we compare our results against the state-of-the-art method V-BM4D [39] with two different choices of parameters. The method labeled V-BM4D-tip corresponds to the results reported in [39]. The results labeled V-BM4D-mp were obtained using the implementation available online [40].…”

“…The first four were chosen because they were used in [39]. The fifth one, football, was added to test the performance of our method on a sequence with complex motions.…”

“…The following These times were computed by averaging the running times obtained for 5 color test sequences for several noise levels. For comparison we also show the running time of V-BM4D [39] using the implementation available online [40] with the "modified profile", which is the one with the highest complexity.…”

“…In [38,39] the authors introduced V-BM4D, an extension of BM3D to video by collaborative filtering of similar motion compensated 3D patches. Trajectories are computed with a block matching strategy based on the sum of squared differences (SSD) and a temporal regularization term, which favours trajectories with small velocity and low acceleration.…”