Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through time

Abstract

Modern neural interfaces allow access to the activity of up to a million neurons within brain circuits. However, bandwidth limits often create a trade-off between greater spatial sampling (more channels or pixels) and the temporal frequency of sampling. Here we demonstrate that it is possible to obtain spatio-temporal super-resolution in neuronal time series by exploiting relationships among neurons, embedded in latent low-dimensional population dynamics. Our novel neural network training strategy, selective backpropagation through time (SBTT), enables learning of deep generative models of latent dynamics from data in which the set of observed variables changes at each time step. The resulting models are able to infer activity for missing samples by combining observations with learned latent dynamics. We test SBTT applied to sequential autoencoders and demonstrate more efficient and higher-fidelity characterization of neural population dynamics in electrophysiological and calcium imaging data. In electrophysiology, SBTT enables accurate inference of neuronal population dynamics with lower interface bandwidths, providing an avenue to significant power savings for implanted neuroelectronic interfaces. In applications to two-photon calcium imaging, SBTT accurately uncovers high-frequency temporal structure underlying neural population activity, substantially outperforming the current state-of-the-art. Finally, we demonstrate that performance could be further improved by using limited, highbandwidth sampling to pretrain dynamics models, and then using SBTT to adapt these models for sparsely-sampled data.

Dynamical flexible inference of nonlinear latent structures in neural population activity

“…Given models trained on fully observed neural observations, we inferred latent factors in the test set that had missing observations. SAE did so by imputing missing observations in the test set to zero as done previously 33,34 , whereas DFINE did so through its new flexible inference method. We then used the inferred factors in the test set to predict behavior variables.…”

“…Because SAE's decoder network is modeled with an RNN, it is structured to take neural observation inputs at every time-step. To handle missing observations for SAE at inference, we imputed them with zeros in the test set as previously done 33,34 , extracted the latent factors, and computed the associated behavior prediction accuracy. DFINE outperformed SAE and, interestingly, this improvement grew larger at lower observed datapoint ratios (Supplementary Fig.…”

“…In contrast, DFINE allows for recursive inference without the need to re-compute any of the past or current factors. In terms of addressing missing observations, while imputation techniques such as zero-padding have been used for SAEs or in general for models with RNNs 33,34 , it is known that such techniques usually yield suboptimal performance 35,36 . This is because imputing missing observations with zero in inference distorts the observation values.…”

“…In a control analysis, we also perform inference with SAE in the presence of missing observations as described above. For SAEs, we impute the missing observations with zeros 33,34 since SAE's generator RNN is designed to take inputs at every time-step. Given the SAE models that are trained on fully observed training sets, we infer the latent factors in the test sets with missing observations, predict behavior variables with learned MLP models and compute the cross-validated behavior prediction accuracy.…”

“…Further, in these models, inference in the presence of missing observations is not directly addressed 4,13,[30][31][32] and using zeros instead of missing observations 33,34 can yield sub-optimal performance by changing the inherent values of missing observations during inference 35,36 (see Results and Discussion). Similarly to these generative models, predictive dynamical models of neural activity that use forward recurrent neural networks [37][38][39] (RNNs) also do not enable the flexible inference properties above.…”