Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit

Abstract

A fundamental challenge in fluorescence microscopy is the photon shot noise arising from the inevitable stochasticity of photon detection. Noise increases measurement uncertainty and limits imaging resolution, speed and sensitivity. To achieve high-sensitivity fluorescence imaging beyond the shot-noise limit, we present DeepCAD-RT, a self-supervised deep learning method for real-time noise suppression. Based on our previous framework DeepCAD, we reduced the number of network parameters by 94%, memory consumption by 27-fold and processing time by a factor of 20, allowing real-time processing on a two-photon microscope. A high imaging signal-to-noise ratio can be acquired with tenfold fewer photons than in standard imaging approaches. We demonstrate the utility of DeepCAD-RT in a series of photon-limited experiments, including in vivo calcium imaging of mice, zebrafish larva and fruit flies, recording of three-dimensional (3D) migration of neutrophils after acute brain injury and imaging of 3D dynamics of cortical ATP release. DeepCAD-RT will facilitate the morphological and functional interrogation of biological dynamics with a minimal photon budget.

Statistically unbiased prediction enables accurate denoising of voltage imaging data

“…The major difference between SUPPORT and DeepCAD-RT 7 or DeepInterpolation 8 , which can also denoise functional imaging data through self-supervised learning, is that DeepCAD-RT and DeepInterpolation learn to predict a frame given temporally adjacent other frames, whereas SUPPORT learns to predict each pixel value by exploiting the information available from both temporally adjacent frames and spatially adjacent pixels in the same time frame. When the imaging speed is not sufficiently faster than the dynamics in the scene (Fig.…”

“…We applied SUPPORT, DeepCAD-RT 7 , and penalized matrix decomposition (PMD) 6 to the synthetic datasets and compared the results. The signals were separated from the backgrounds in the denoised videos (Methods) to compare their accuracy in recovering the time-varying signal (Fig.…”

“…Both DeepCAD-RT 7 and DeepInterpolation 8 are based on the assumption that the underlying signal in any two consecutive frames in a video can be considered the same, whereas the noise is independent when the imaging speed is sufficiently higher than the dynamics of the fluorescent reporter 7,8 ; the networks are trained to predict the “current” frame using the past and future frames as the input. Unfortunately, this assumption breaks down when the imaging speed is not sufficiently faster than the dynamics, and the bias in the denoising outcome is increased.…”

“…The decrease in SNR not only hinders the accurate detection of the neurons' location but also compromises the timing precision of the detected temporal events, which nullifies the increase in temporal resolution. Fortunately, all functional imaging data have high inherent redundancy in the sense that each frame in a dataset is nearly identical to other frames apart from noise, which offers an opportunity to denoise or distinguish the signal from the noise in the data [6][7][8] .Denoising is a type of signal processing that attempts to extract underlying signals from noisy observations based on prior knowledge of the signal and the noise 9 . Importantly, the fundamental property of noise-randomness-does not allow for exact recovery of the signal, so we can only reduce statistical variance at the cost of increasing statistical bias (i.e., an absolute deviation between the mean denoising outcome and the ground truth).…”

“…The high representational power of convolutional networks allows for learning nearly arbitrary signal models in the image domain, which result in low bias in denoising outcomes without sacrificing variance 18 . Owing to its high representational power and the high inherent redundancy in functional imaging data, convolutional networks have shown enormous success in denoising functional imaging data 7,8 . As a key aspect, these methods learn the signal model from noisy data in a self-supervised manner [19][20][21][22] , so the need for "clean" images is alleviated.Both DeepCAD-RT 7 and DeepInterpolation 8 are based on the assumption that the underlying signal in any two…”