High spatio-temporal resolution video with compressed sensing

Koller, Roman; Schmid, Lukas; Matsuda, Nathan; Niederberger, Thomas; Spinoulas, Leonidas; Cossairt, Oliver; Schuster, G.M.; Katsaggelos, Aggelos K.

doi:10.1364/oe.23.015992

Cited by 76 publications

(48 citation statements)

References 35 publications

(60 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another technique, proposed by Llull and Yuan et al, achieved high-speed video reconstruction (22 frames at 660 fps) from a single-shot coded-aperture image that is obtained by translating binary amplitude masks within the focal plane of a global shutter sensor [3], [4]. Koller et al later improved the mask design [5] and Liu et al proposed a reconstruction that exploits the low-rank structure of the underlying scene [6]. The commonality between these setups is that each pixel is temporally modulated during the exposure, and all require bulky and expensive hardware.…”

Section: Related Workmentioning

confidence: 99%

Video from Stills: Lensless Imaging with Rolling Shutter

Antipa

Oare

Bostan

et al. 2019

2019 IEEE International Conference on Computational Photography (ICCP)

View full text Add to dashboard Cite

Because image sensor chips have a finite bandwidth with which to read out pixels, recording video typically requires a trade-off between frame rate and pixel count. Compressed sensing techniques can circumvent this trade-off by assuming that the image is compressible. Here, we propose using multiplexing optics to spatially compress the scene, enabling information about the whole scene to be sampled from a row of sensor pixels, which can be read off quickly via a rolling shutter CMOS sensor. Conveniently, such multiplexing can be achieved with a simple lensless, diffuser-based imaging system. Using sparse recovery methods, we are able to recover 140 video frames at over 4,500 frames per second, all from a single captured image with a rolling shutter sensor. Our proof-of-concept system uses easily-fabricated diffusers paired with an off-the-shelf sensor. The resulting prototype enables compressive encoding of high frame rate video into a single rolling shutter exposure, and exceeds the sampling-limited performance of an equivalent global shutter system for sufficiently sparse objects.

show abstract

Section: Related Workmentioning

confidence: 99%

Video from Stills: Lensless Imaging with Rolling Shutter

Antipa

Oare

Bostan

et al. 2019

2019 IEEE International Conference on Computational Photography (ICCP)

View full text Add to dashboard Cite

show abstract

“…In recent work, Koller et al . [KSM*15] have performed an analysis of several state‐of‐the‐art approaches for high spatiotemporal resolution video reconstruction with compressed sensing. They prove that the approach proposed by Liu et al .…”

Section: Resultsmentioning

confidence: 99%

Convolutional Sparse Coding for Capturing High‐Speed Video Content

Serrano¹,

Garcés²,

Gutiérrez³

et al. 2017

Computer Graphics Forum

View full text Add to dashboard Cite

single-coded image and a trained dictionary of image patches. In this paper, we first analyse this approach, and find insights that help improve the quality of the reconstructed videos. We then introduce a novel technique, based on convolutional sparse coding (CSC), and show how it outperforms the state-of-the-art, patch-based approach in terms of flexibility and efficiency, due to the convolutional nature of its filter banks. The key idea for CSC high-speed video acquisition is extending the basic formulation by imposing an additional constraint in the temporal dimension, which enforces sparsity of the first-order derivatives over time.

show abstract

“…This method could spare the decoding for the static area in the scene, but not meet the conditions for random coding. Roman Koller et al [8] provided the ShiftedNormalized coding method, which combined the advantages of Llull's and Liu's methods [16]. For the same time periods of sampling, the luminous flux would remain the same for all pixels; the technique could be realized by the mask or liquid crystal light valve.…”

Section: Forward Model For Cactimentioning

confidence: 99%

MAP-MRF-Based Super-Resolution Reconstruction Approach for Coded Aperture Compressive Temporal Imaging

Zhang

Gao

2018

Applied Sciences

View full text Add to dashboard Cite

Coded Aperture Compressive Temporal Imaging (CACTI) can afford low-cost temporal super-resolution (SR), but limits are imposed by noise and compression ratio on reconstruction quality. To utilize inter-frame redundant information from multiple observations and sparsity in multi-transform domains, a robust reconstruction approach based on maximum a posteriori probability and Markov random field (MAP-MRF) model for CACTI is proposed. The proposed approach adopts a weighted 3D neighbor system (WNS) and the coordinate descent method to perform joint estimation of model parameters, to achieve the robust super-resolution reconstruction. The proposed multi-reconstruction algorithm considers both total variation (TV) and 2,1 norm in wavelet domain to address the minimization problem for compressive sensing, and solves it using an accelerated generalized alternating projection algorithm. The weighting coefficient for different regularizations and frames is resolved by the motion characteristics of pixels. The proposed approach can provide high visual quality in the foreground and background of a scene simultaneously and enhance the fidelity of the reconstruction results. Simulation results have verified the efficacy of our new optimization framework and the proposed reconstruction approach.

show abstract

High spatio-temporal resolution video with compressed sensing

Cited by 76 publications

References 35 publications

Video from Stills: Lensless Imaging with Rolling Shutter

Video from Stills: Lensless Imaging with Rolling Shutter

Convolutional Sparse Coding for Capturing High‐Speed Video Content

MAP-MRF-Based Super-Resolution Reconstruction Approach for Coded Aperture Compressive Temporal Imaging

Contact Info

Product

Resources

About