Robust Video Restoration by Joint Sparse and Low Rank Matrix Approximation

Abstract

Abstract. This paper presents a new video restoration scheme based on the joint sparse and lowrank matrix approximation. By grouping similar patches in the spatiotemporal domain, we formulate the video restoration problem as a joint sparse and low-rank matrix approximation problem. The resulted nuclear norm and 1 norm related minimization problem can also be efficiently solved by many recently developed numerical methods. The efficiency of the proposed video restoration scheme is illustrated on two applications: video denoising in the presence of random-valued noise and video in-painting for archived films. The numerical experiments indicated the proposed video restoration method is compared favorably against many existing algorithms.Key words. nuclear norm, low-rank matrix, sparse matrix, denoising, in-painting AMS subject classifications. 68U10, 65J22, 90C25, 65K051. Introduction. Even with today's advances in camera and digital sensor technology, video data collected in practice often suffers from many types of annoying degradations, e.g., noise contamination, image blurring and missing data. For example, video data can be quite noisy when captured at high sensitivities, such as low lighting condition, high ISO setting or high capture rate. The frames of video data can be blurred when there are fast moving objects in the scene or when there are camera shakes. Some parts of video data can be missing or invisible from a user perspective due to occlusions, scratches, or errors in data conversion/communication. The goal of video restoration is then to recover the original one from the obtained degraded video data. With the prevalence of webcams and camera phones, the problem of video restoration has become even more important than before.In recent years, patch-based image restoration scheme has emerged as one promising approach for various image restoration tasks, e.g. denoising and in-painting ([1, 2, 3]). The basic idea of these approaches is to regularize the restoration process by utilizing the spatial redundancy of original static images. Compared to static image data, video data tends to be of lower quality due to the high speed capturing rate of video camera, e.g. lower signal-to-noise ratio and lower image resolution. Meanwhile, owing to its significant temporal redundancy, video data usually provides much richer information about the scene than static image data does. Thus, the efficiency of restoring degraded video data largely depends on how efficient the temporal redundancy is utilized in the methods. Some of patched based image restoration methods have been extended to the case of video denoising ([4, 5]). Although differing from details, these video restoration methods are built upon the same methodology that exploits the self-similarity in video by examining the the similarity among different image patches.The aforementioned patch-based image/video restoration methods showed impressive results on suppressing image noise when the noise is mostly Gaussian white noise. In practice, the existence of ou...

Single-Patch Low-Rank Prior for Non-pointwise Impulse Noise Removal

“…A joint low-rank and sparse matrix recovery framework was recently used to detect and remove impulse noise simultaneously [11], remove background, and remove shadows and specularities from face images [3]. However, the method is limited by the usage of multiple similar patches and the small size of patches.…”

“…When a rectangular gray image patch P contains random Gaussian noise and RVIN, P may be decomposed as P = L * + S * + N * , where L * represents the unknown noise-free patch, S * represents the unknown impulse noise (precisely, the difference between the noise and the corresponding noise-free pixel values), and N * is a matrix of Gaussian noise [11]. L * can be considered a low-rank matrix due to the lowrank prior for single patches (Section 6.1).…”

“…Also, since the number of pixels corrupted by impulse noise is generally much smaller than the total number of pixels, S * is a sparse matrix. As a result, the problem of image denoising can be formulated as an optimization problem [11], i.e., to minimize E 1 (L, S) over the matrix variables L and S,…”

“…While the above optimization framework has been used for image and video denoising [11], each image patch was considered as a column in a matrix. Such a multi-patch method is subject to the limitations discussed in Section 1.…”

“…Based on the observation that almost any properly-oriented (defined later) small noise-free image patch (being a matrix) can be approximated by a lowrank patch with texture details well preserved, we propose a single-patch method to simultaneously detect and remove RVIN within a generalized joint low-rank and sparse matrix recovery framework. While the original matrix recovery framework has been recently used for image and video denoising [11], it requires multiple similar image patches, with each patch vectorized as a column in the matrix. Such a multi-patch method often limits the size of image patches to be relatively small (e.g., 8×8 pixels) in order to more likely find multiple similar patches within a single image and to collaboratively and effectively remove traditional (singlepixel) RVIN.…”