Unsupervised learning and clustered connectivity enhance reinforcement learning in spiking neural networks

Abstract

2Reinforcement learning is a learning paradigm that can account for how organisms learn to 3 adapt their behavior in complex environments with sparse rewards. However, implementations in 4 spiking neuronal networks typically rely on input architectures involving place cells or receptive 5 fields. This is problematic, as such approaches either scale badly as the environment grows in size 6 or complexity, or presuppose knowledge on how the environment should be partitioned. Here, we 7 propose a learning architecture that combines unsupervised learning on the input projections with 8 clustered connectivity within the representation layer. This combination allows input features to 9 be mapped to clusters; thus the network self-organizes to produce task-relevant activity patterns 10 that can serve as the basis for reinforcement learning on the output projections. On the basis 11 of the MNIST and Mountain Car tasks, we show that our proposed model performs better than 12 either a comparable unclustered network or a clustered network with static input projections. We 13 conclude that the combination of unsupervised learning and clustered connectivity provides a 14 generic representational substrate suitable for further computation. 15

Face detection in untrained deep neural networks

“…It is notable that recent studies suggest that the random network can perform this task if the read-out units are selected via a prior understanding of the system. For example, object classification can be performed by a random network if read-outs are chosen by a synaptic rule observed in the brain 85 , 86 . While these models focus on the innate functions of networks in that a high dimensional space generated by a random network can perform various tasks without learning, our current results demonstrate that the functional tuning of single units (comparable to neuronal tuning in biological brains) can arise in random networks without any further training of the read-out process, which is distinguished from the main idea of the reservoir computing model.…”

Covariance-based information processing in reservoir computing systems

“…Overall, there is a nontrivial interaction between reservoir topology and input structure which should be further investigated. We note that we explored naive connectivities within the segregated reservoir and topologies displaying more biologically realistic traits, like small-world, scale-free, clustered (Weidel et al, 2020) or even real connectome patterns (Suárez et al, 2020; Damicelli et al, 2021), could be investigated. Furthermore, we constrained our study to random input projections, which take part in shaping the input representations that arise in the reservoir.…”

“…Furthermore, we constrained our study to random input projections, which take part in shaping the input representations that arise in the reservoir (as hinted in Section 2.1.1). Along this line, it has been shown that unsupervised plasticity at the input-to-reservoir layer improves performance in pattern recognition tasks with mean-based decoding (Weidel et al, 2020), so it is natural to question whether this effect is also observed, or even further enhanced, for covariance-based readouts.…”

Long- and short-term history effects in a spiking network model of statistical learning

“…In both models, the architecture is identical, leading to comparable design features: for example, the accuracy and speed of learning depend to a large extent on the number of clusters in the backbone. A clustered connectivity has also been shown to enhance reinforcement learning in a recent study (Weidel et al 2021). More generally, a clustered code has interesting properties, such as robust error correction, that make it a candidate to underlie computations in the brain (Berry and Tkačik 2020).…”