Adaptive Deep PnP Algorithm for Video Snapshot Compressive Imaging

Abstract

Video Snapshot compressive imaging (SCI) is a promising technique to capture high-speed videos, which transforms the imaging speed from the detector to mask modulating and only needs a single measurement to capture multiple frames. The algorithm to reconstruct high-speed frames from the measurement plays a vital role in SCI. In this paper, we consider the promising reconstruction algorithm framework, namely plug-andplay (PnP), which is flexible to the encoding process comparing with other deep learning networks. One drawback of existing PnP algorithms is that they use a pretrained denoising network as a plugged prior while the training data of the network might be different from the task in real applications. Towards this end, in this work, we propose the online PnP algorithm which can adaptively update the network's parameters within the PnP iteration; this makes the denoising network more applicable to the desired data in the SCI reconstruction. Furthermore, for color video imaging, RGB frames need to be recovered from Bayer pattern or named demosaicing in the camera pipeline. To address this challenge, we design a two-stage reconstruction framework to optimize these two coupled ill-posed problems and introduce a deep demosaicing prior specifically for video demosaicing. Extensive results on both simulation and real datasets verify the superiority of our adaptive deep PnP algorithm. The code to reproduce the results is at https: //github.com/xyvirtualgroup/AdaptivePnP_SCI.

An Overview on Deep Learning Techniques for Video Compressive Sensing

“…We present a quantitative comparison to compare the quality performances of the following VCS algorithms: GAP-TV [44], DeSCI [73], PnP-FFDNet [74], Pnp-FastDVDNet [88], GAP-FastDVDNet(online) [85], DE-RNN [86], DE-GAP-FFDnet [86], E2E-CNN [48], BIR-NAT [75], MetaSCI [83], RevSCI [82], DeepUnfold-VCS [51], GAP-Unet-S12 [76], ELP-Unfolding [84].…”

“…Recently, an ensemble learning based algorithm is proposed in [84], originally exploited in inverse problems, to enhance the scalability of video SCI reconstruction approaches. Zongliang et al [85] still work on combining iterative algorithms and deep neural networks. An online Plug-and-play algorithm is proposed to adaptively update the model's parameters using the PnP iteration, which enhance the network's noise resistance.…”

Recent advances of deep learning for spectral snapshot compressive imaging

“…(a) Algorithms based on iterative optimization , whose design assume that the source structure is captured by (typically simple) priors (or regularization functions), such as sparsity 47 or total variation (TV) ; (c) Deep unfolding algorithms, where, instead of one CNN, a sequence of K CNNs are trained to map measurements to the desired signal; 9, 11, 29, 30 unlike E2E-CNNs, here, each stage consists of two parts: i) linear transformation and ii) passing the signal through a CNN operating as a denoiser, and (d) Plug-and-play (PnP) methods 51,52 in which a pre-trained denoising network is plugged into one of the iterative algorithms of class (a). 27,[53][54][55] Note that all these deep learning based algorithms are in the supervised fashion, which means they need a large amount of training data to train the network to get good results. This pose limitation to systems that are challenging to get rich training data such as the endoscopy.…”

Deep-learning enables single-pixel spectral imaging

“…[14][15][16] The combination of deep learning and conventional iterative algorithms gives birth to the state-of-the-art reconstruction methods dubbed deep unfolding network [17][18][19] and the more flexible, plug-and-play algorithms. [20][21][22] And recently, Transformer architecture has also been introduced into image restoration to replace convolution neural network (CNN) with surprising results. [23][24][25][26][27][28] Here, we adopt Controllable Arbitrary-Sampling Network (COAST) 29 and Plug-and-play generalized alternating projection (PnP-GAP) 30,31 with Convolutional-Neural-Network-based (CNN-based) deep spectral denoising prior 32 to solve this inverse problem of CACS-SPI's reconstruction in simulation.…”