Differentiable Rendering for 3D Fluorescence Microscopy

Abstract

Differentiable rendering is a growing field that is at the heart of many recent advances in solving inverse graphics problems, such as the reconstruction of 3D scenes from 2D images. By making the rendering process differentiable, one can compute gradients of the output image with respect to the different scene parameters efficiently using automatic differentiation. Interested in the potential of such methods for the analysis of fluorescence microscopy images, we introduce deltaMic, a microscopy renderer that can generate a 3D fluorescence microscopy image from a 3D scene in a fully differentiable manner. By convolving the meshes in the scene with the point spread function (PSF) of the microscope, that characterizes the response of its imaging system to a point source, we emulate the 3D image creation process of fluorescence microscopy. This is achieved by computing the Fourier transform (FT) of the mesh and performing the convolution in the Fourier domain. Naive implementation of such mesh FT is however slow, inefficient, and sensitive to numerical precision. We solve these difficulties by providing a memory and computationally efficient fully differentiable GPU implementation of the 3D mesh FT. We demonstrate the potential of our method by reconstructing complex shapes from artificial microscopy images. Eventually, we apply our renderer to real confocal fluorescence microscopy images of embryos to accurately reconstruct the multicellular shapes of these cell aggregates.

Embryo mechanics cartography: inference of 3D force atlases from fluorescence microscopy

“…A possible generic avenue to solve these problems may lie in a fully variational approach, where a mathematical loss between the microscopy images and the meshes could be constrained by an arbitrary mechanical model to allow direct gradientbased optimization of its spatio-temporal parameters. Our recent effort to design such an efficient loss for comparing a mesh and an image may begin to fill this gap [83]. Importantly, the current force inference method we introduced will remain a fundamental building block to this research field, providing already accurate geometric and mechanical maps, which will form an ideal initial guess to refined but more computationally expensive iterative methods.…”

3d Monte Carlo Differentiable Radiative Transfer with Dart

“…Jakob et al (2022) and Vicini (2022) improved computation efficiency with respect to scene parameter optimization. Present research on differentiable renderers is mostly focused on reconstructing the shape and appearance of real-world objects from very high resolution RGB images (Ichbiah et al, 2023;Jiang et al, 2020;Luan et al, 2021;Petersen et al, 2022). Salesin et al (2024) will adapt these methods for retrieving aerosol size and refractive index in a simple atmosphere-ocean scene in the future work.…”