Heterogeneity in Neuronal Dynamics is Learned by Gradient Descent for Temporal Processing Tasks

Abstract

Individual neurons in the brain have complex intrinsic dynamics that are highly diverse. We hypothesize that the complex dynamics produced by networks of complex and heterogeneous neurons may contribute to the brain's ability to process and respond to temporally complex data. To study the role of complex and heterogeneous neuronal dynamics in network computation, we develop a rate-based neuronal model, the generalized-leaky-integrate-and-firing-rate (GLIFR) model, which is a rate-equivalent of the generalized-leaky-integrate-and-fire model. The GLIFR model has multiple dynamical mechanisms which add to the complexity of its activity while maintaining differentiability. We focus on the role of after-spike currents, currents induced or modulated by neuronal spikes, in producing rich temporal dynamics. We use machine learning techniques to learn both synaptic weights and parameters underlying intrinsic dynamics to solve temporal tasks. The GLIFR model allows us to use standard gradient descent techniques rather than surrogate gradient descent, which has been utilized in spiking neural networks. After establishing the ability to optimize parameters using gradient descent in single neurons, we ask how networks of GLIFR neurons learn and perform on temporally challenging tasks, such as sinusoidal pattern generation and sequential MNIST. We find that these networks learn a diversity of parameters, which gives rise to diversity in neuronal dynamics. We also observe that training networks on the sequential MNIST task leads to formation of cell classes based on the clustering of neuronal parameters. GLIFR networks have mixed performance when compared to vanilla recurrent neural networks but appear to be more robust to random silencing. When we explore these performance gains further, we find that both the ability to learn heterogeneity and the presence of after-spike currents contribute. Our work both demonstrates the computational robustness of neuronal complexity and diversity in networks and demonstrates a feasible method of training such models using exact gradients.

Neural networks with optimized single-neuron adaptation uncover biologically plausible regularization

“…Furthermore, we observe that when the activation function γ is initialized homogeneously, the optimization procedure leads to heterogeneity in the activation functions across the network (Fig.3 a top). See Winston et al (2022a) for similar results when AFs are parametrized following known relations between ionic currents and f-I curves. Further experiments (details included in Appendix §B.2) consider trained RNN+ γ networks reading psMNIST digits rotated by π/ 4 rad.…”

“…3a top). See Winston et al (2022a) for similar results when AFs are parametrized following known relations between ionic currents and f-I curves. Further experiments (details included in Appendix §B.2) consider trained RNN+γ networks reading psMNIST digits rotated by π/4 rad.…”

“…This is shared approach that yields impressive results across varied settings. Notably, Winston et al (2022b) similarly optimize static neuron properties in RNNs and find advantages of diversity, and the line of work spawned by Yamins and DiCarlo (2016) keeps reveling stimulus tuning properties of neurons in the visual pathway and beyond.…”

“…At the network level, adaptive neural responses have been shown to support efficient coding with metabolic advantages (Gutierrez and Denève, 2019), facilitate computations over long timescales (Bellec et al, 2018; Fitz et al, 2020; Salaj et al, 2021), and even enable forms of Bayesian inference (Deneve, 2008; Kilpatrick and Ermentrout, 2011). Recent work also shows robustness gains from learned modulated neural dynamics (Vecoven et al, 2020) and with diverse and dynamics synapses and IF curves (Burnham et al, 2021; Winston et al, 2022a), While a number of coding advantages of diverse and dynamic single neuron responses are now established, it is still unknown how these mechanisms have come to bear, and how they influence learning and configuration of larger neural networks that support system-level tasks such as perception or prediction.…”

Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules

“…To address our main question, this study focuses on the effect of various learning rules while holding data, objective function and architecture constant (see [10]). However, these different components can interact, and more sophisticated architecture can facilitate task learning [154][155][156][157][158][159][160][161]. Given the exploding parameter space resulting from such interactions, we believe it requires careful future analysis and is outside of the scope for this one paper.…”