Real-Time Light Field 3D Microscopy via Sparsity-Driven Learned Deconvolution

Vizcaíno, Josué Page; Wang, Zeguan; Symvoulidis, Panagiotis; Favaro, Paolo; Guner-Ataman, Burcu; Boyden, Edward S.; Lasser, Tobias

doi:10.1109/iccp51581.2021.9466256

Cited by 5 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main factor that limits the speed of the system is the time it takes to reconstruct the volumetric image of the fish that is used to register in the ZBB atlas. Using deep learning algorithms (41)(42)(43)(44) instead of the traditional Richardson-Lucy iterative reconstruction method is expected to greatly accelerate the speed of reconstruction and make our system more responsive to rapid and high-frequency head movements.…”

Section: Discussionmentioning

confidence: 99%

All-optical interrogation of brain-wide activity in freely swimming larval zebrafish

Chai

et al. 2023

Preprint

View full text Add to dashboard Cite

We introduce an all-optical technique that enables volumetric imaging of brain-wide calcium activity and targeted optogenetic stimulation of specific brain regions in freely swimming larval zebrafish. The system consists of three main components: a 3D tracking module, a dual color fluorescence imaging module, and a real-time activity manipulation module. Our approach uses a sensitive genetically encoded calcium indicator in combination with a long Stokes shift red fluorescence protein as a reference channel, allowing the extraction of Ca2+ activity from signals contaminated by motion artifacts. The method also incorporates rapid 3D image reconstruction and registration, facilitating real-time selective optogenetic stimulation of different regions of the brain. By demonstrating that selective light activation of the midbrain regions in larval zebrafish could reliably trigger biased turning behavior and an asymmetric propagation of brain-wide neural activity, we present a valuable tool for investigating the causal relationship between distributed neural circuit dynamics and naturalistic behavior.

show abstract

Section: Discussionmentioning

confidence: 99%

All-optical interrogation of brain-wide activity in freely swimming larval zebrafish

Chai

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, machine learning models for light field depth estimation have been extensively researched and vary widely. Focusing on the trade-off between accuracy and speed, for instance, Vizcaino et al [16] proposed a model that is fast and capable of real-time processing, whereas Jeon et al [17] presented a model that prioritizes high-precision depth estimation. When constructing these networks, feature extraction type is the crucial for machine learning.…”

Section: Light Field Cameramentioning

confidence: 99%

Balancing Speed and Precision: Lightweight and Accurate Depth Estimation for Light Field Image

Miya,

Kawaguchi,

Saito

2024

Preprint

View full text Add to dashboard Cite

With the progression of AI, embedding advanced AI technologies into small robotics and mobile devices has become essential, driving research towards lightweighting AI models. Our study enhances the EPINET depth estimation model for light field images, aiming for compactness and faster inference while preserving accuracy. We conducted two-step experiments aimed at enhancing inference efficiency: Initially, by adjusting input streams and convolution layers, we simplified the CNN model, achieving faster inference times at the cost of reduced accuracy. To address this reduction in accuracy, we then applied knowledge distillation, allowing the simplified model to learn from the original model’s more complex patterns. In our quantitative experiments using two error metrics, MSE (Mean Squared Error) and BadPix, we identified optimal knowledge positions and evaluated the required complexity for the student model. As a result, our method improved MSE by 21% and BadPix by 14% compared to training without it. Furthermore, the student model achieved an inference speed 13% faster than the teacher model and surpassed its accuracy by 10% in MSE. Additionally, we demonstrated that repeatedly applying our approach could further enhance both model compactness and accuracy.

show abstract

“…The lightfield scattered or emitted by the sample can be computationally retrieved through different approaches. The 3D reconstruction of the sample can be computed based on back-propagation algorithms [25], deconvolution-based techniques [26,27], or even deep learning methodologies [28].…”

Section: Introductionmentioning

confidence: 99%

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy

Rostan

Incardona

Sánchez-Ortiga

et al. 2022

Sensors

View full text Add to dashboard Cite

Current interest in Fourier lightfield microscopy is increasing, due to its ability to acquire 3D images of thick dynamic samples. This technique is based on simultaneously capturing, in a single shot, and with a monocular setup, a number of orthographic perspective views of 3D microscopic samples. An essential feature of Fourier lightfield microscopy is that the number of acquired views is low, due to the trade-off relationship existing between the number of views and their corresponding lateral resolution. Therefore, it is important to have a tool for the generation of a high number of synthesized view images, without compromising their lateral resolution. In this context we investigate here the use of a neural radiance field view synthesis method, originally developed for its use with macroscopic scenes acquired with a moving (or an array of static) digital camera(s), for its application to the images acquired with a Fourier lightfield microscope. The results obtained and presented in this paper are analyzed in terms of lateral resolution and of continuous and realistic parallax. We show that, in terms of these requirements, the proposed technique works efficiently in the case of the epi-illumination microscopy mode.

show abstract

Real-Time Light Field 3D Microscopy via Sparsity-Driven Learned Deconvolution

Cited by 5 publications

References 22 publications

All-optical interrogation of brain-wide activity in freely swimming larval zebrafish

All-optical interrogation of brain-wide activity in freely swimming larval zebrafish

Balancing Speed and Precision: Lightweight and Accurate Depth Estimation for Light Field Image

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy

Contact Info

Product

Resources

About