3D Localization for Light-Field Microscopy via Convolutional Sparse Coding on Epipolar Images

Abstract

Light-field microscopy (LFM) is a type of all-optical imaging system that is able to capture 4D geometric information of light rays and can reconstruct a 3D model from a single snapshot. In this paper, we propose a new 3D localization approach to effectively detect 3D positions of neuronal cells from a single light-field image with high accuracy and outstanding robustness to light scattering. This is achieved by constructing a depth-aware dictionary and by combining it with convolutional sparse coding. Specifically, our approach includes 3 key parts: light-field calibration, depth-aware dictionary construction, and localization based on convolutional sparse coding (CSC). In the first part, an observed raw light-field image is calibrated and then decoded into a two-plane parameterized 4D format which leads to the epi-polar plane image (EPI). The second part involves simulating a set of light-fields using a wave-optics forward model for a ball-shaped volume that is located at different depths. Then, a depth-aware dictionary is constructed where each element is a synthetic EPI associated to a specific depth. Finally, by taking full advantage of the sparsity prior and shift-invariance property of EPI, 3D localization is achieved via convolutional sparse coding on an observed EPI with respect to the depthaware EPI dictionary. We evaluate our approach on both nonscattering specimen (fluorescent beads suspended in agarose gel) and scattering media (brain tissues of genetically encoded mice). Extensive experiments demonstrate that our approach can reliably detect the 3D positions of granular targets with small Root Mean Square Error (RMSE), high robustness to optical aberration and light scattering in mammalian brain tissues.

Subcellular resolution three-dimensional light-field imaging with genetically encoded voltage indicators

“…Deconvolving individual frames with these sensors would require a drastic increase in computational resources and is likely untenable. Source extraction approaches have been developed for static light-field images 89 and light-field calcium imaging time series, 65,90 which do not involve deconvolution of every frame. In their current form, however, these are unsuitable for reconstruction of subcellular light-field voltage imaging time series as they leverage the temporal and/or spatial characteristics of neuronal calcium imaging as reconstruction priors.…”

Comparing synthetic refocusing to deconvolution for the extraction of neuronal calcium transients from light-fields

“…Despite this, all the cells in this study were relatively superficial, within the photon mean free path. Methods to improve performance in scattering tissue have been demonstrated by computationally extracting fluorescence sources [13,17,19,35], or by combining the principles of confocal microscopy with LFM [36].…”

Model-Inspired Deep Learning for Light-Field Microscopy with Application to Neuron Localization

“…In this section, we evaluate the performance of CISTA-net on the 3D localization task. We also compare our approach with the phase-space based method (Phase-Space for short) [6,8] and convolutional sparse coding based method (CSC for short) [9] on light-field microscopy data obtained from scattering specimens -genetically encoded fluorophore in mouse brain tissues, as shown in Fig. 5 (a).…”

“…The method developed in [9] leverages the fact that neurons localized in space are relatively similar to point-like sources. Given raw light-field microscopy images, we use the method proposed in [9] to perform calibration, conversion, and purification to get an array of clean sub-aperture images. Then, the horizontal and vertical positions, i.e.…”

“…Such an EPI dictionary can be used to decompose an input EPI into a sparse linear combination of basic EPIs. Finding this sparse decomposition can be modelled as a convolutional sparse coding problem (1)(See also [9]). Specifically, given an observed EPI X ∈ R Θ×N where Θ denotes angular dimension, i.e.…”