Comparing Metaheuristic Algorithms on the Training Process of Spiking Neural Networks

Espinal, Andrés; Carpío, Martín; Ornelas, Manuel; Soberanes, Héctor José Puga; Melín, Patricia; Sotelo-Figueroa, Marco Aurelio

doi:10.1007/978-3-319-05170-3_27

Cited by 5 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditionally, when SNN models are trained to perform a behavior using biologically inspired learning mechanisms, algorithms used are variations on either spike timing dependent plasticity(STDP) (Tavanaei et al, 2019) or evolutionary strategies (EVOL) (Espinal et al, 2014). For learning behaviors from the reinforcement learning domain, STDP can be extended to use reward modulated plasticity (Chadderdon et al, 2012;Neymotin et al, 2013;Hazan et al, 2018;Patel et al, 2019;Anwar et al, 2022a), an algorithm denoted spike-timing dependent reinforcement learning (STDP-RL).…”

Section: Introductionmentioning

confidence: 99%

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Hasegan¹,

Deible

Earl

et al. 2022

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Artificial neural networks (ANNs) have been successfully trained to perform a wide range of sensory-motor behaviors. In contrast, the performance of spiking neuronal network (SNN) models trained to perform similar behaviors remains relatively suboptimal. In this work, we aimed to push the field of SNNs forward by exploring the potential of different learning mechanisms to achieve optimal performance. We trained SNNs to solve the CartPole reinforcement learning (RL) control problem using two learning mechanisms operating at different timescales: (1) spike-timing-dependent reinforcement learning (STDP-RL) and (2) evolutionary strategy (EVOL). Though the role of STDP-RL in biological systems is well established, several other mechanisms, though not fully understood, work in concert during learning in vivo. Recreating accurate models that capture the interaction of STDP-RL with these diverse learning mechanisms is extremely difficult. EVOL is an alternative method and has been successfully used in many studies to fit model neural responsiveness to electrophysiological recordings and, in some cases, for classification problems. One advantage of EVOL is that it may not need to capture all interacting components of synaptic plasticity and thus provides a better alternative to STDP-RL. Here, we compared the performance of each algorithm after training, which revealed EVOL as a powerful method for training SNNs to perform sensory-motor behaviors. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

show abstract

Section: Introductionmentioning

confidence: 99%

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Hasegan¹,

Deible

Earl

et al. 2022

Front. Comput. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Traditionally, when SNN models are trained to perform a behavior using biologically inspired learning mechanisms, algorithms used are variations on either Spike Timing Dependent Plasticity(STDP) (Tavanaei et al 2019) or Evolutionary Strategies (Espinal et al 2014). For learning behaviors from the reinforcement learning domain, STDP can be extended to use reward modulated plasticity (Anwar et al 2021; Patel et al 2019; Hazan et al 2018; Chadderdon et al 2012; Neymotin et al 2013), an algorithm denoted Spike-timing dependent reinforcement learning (STDP-RL).…”

Section: Introductionmentioning

confidence: 99%

Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control

Hasegan

Deible

Earl

et al. 2021

Preprint

View full text Add to dashboard Cite

Biological learning operates at multiple interlocking timescales, from long evolutionary stretches down to the relatively short time span of an individual’s life. While each process has been simulated individually as a basic learning algorithm in the context of spiking neuronal networks (SNNs), the integration of the two has remained limited. In this study, we first train SNNs separately using individual model learning using spike-timing dependent reinforcement learning (STDP-RL) and evolutionary (EVOL) learning algorithms to solve the CartPole reinforcement learning (RL) control problem. We then develop an interleaved algorithm inspired by biological evolution that combines the EVOL and STDP-RL learning in sequence.We use the NEURON simulator with NetPyNE to create an SNN interfaced with the CartPole environment from OpenAI’s Gym. In CartPole, the goal is to balance a vertical pole by moving left/right on a 1-D plane. Our SNN contains multiple populations of neurons organized in three layers: sensory layer, association/hidden layer, and motor layer, where neurons are connected by excitatory (AMPA/NMDA) and inhibitory (GABA) synapses. Association and motor layers contain one excitatory (E) population and two inhibitory (I) populations with different synaptic time constants. Each neuron is an event-based integrate-and-fire model with plastic connections between excitatory neurons. In our SNN, the environment activates sensory neurons tuned to specific features of the game state. We split the motor population into subsets representing each movement choice. The subset with more spiking over an interval determines the action.During STDP-RL, we supply intermediary evaluations (reward/punishment) of each action by judging the effectiveness of a move (e.g., moving the CartPole to a balanced position). During EVOL, updates consist of adding together many random perturbations of the connection weights. Each set of random perturbations is weighted by the total episodic reward it achieves when applied independently. We evaluate the performance of each algorithm after training and through the creation of sensory/motor action maps that delineate the network’s transformation of sensory inputs into higher-order representations and eventual motor decisions. Both EVOL and STDP-RL training produce SNNs capable of moving the cart left and right and keeping the pole vertical. Compared to the STDP-RL and EVOL algorithms operating on their own, our interleaved training paradigm produced enhanced robustness in performance, with different strategies revealed through analysis of the sensory/motor mappings. Analysis of synaptic weight matrices also shows distributed vs clustered representations after the EVOL and STDP-RL algorithms, respectively. These weight differences also manifest as diffuse vs synchronized firing patterns. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

show abstract

“…Other works, in [43, 54–57], three-layered feedforward SNNs with synaptic connections were implemented, which are formed by a weight and a delay, to solve supervised classification problems through the use of time-to-first-spike as a classification criterion; in these works, the training has been carried out by means of evolutionary strategy (ES) [58, 59] and PSO algorithms. An extension of previous works is made in [60, 61], where the number of hidden layers and their computing units are defined by grammatical evolution (GE) [62] besides the metaheuristic learning. More complex SNN frameworks have been developed and trained with metaheuristics (such as ES) to perform tasks such as visual pattern recognition, audio-visual pattern recognition, taste recognition, ecological modelling, sign language recognition, object movement recognition, and EEG spatio/spectrotemporal pattern recognition (see [63] for a review of these frameworks).…”

Section: Introductionmentioning

confidence: 99%

“…More complex SNN frameworks have been developed and trained with metaheuristics (such as ES) to perform tasks such as visual pattern recognition, audio-visual pattern recognition, taste recognition, ecological modelling, sign language recognition, object movement recognition, and EEG spatio/spectrotemporal pattern recognition (see [63] for a review of these frameworks). The robotic locomotion is solved through SNNs designed by metaheuristics in [60, 64, 65]; in these works, both the connectivity pattern and synaptic weights of each Belson–Mazet–Soula (BMS) [66] neuron model into SNNs called spiking central pattern generators (SCPGs) are defined through GE or Christiansen grammar evolution (CGE) [67] algorithms; all individual designs are integrated to define the SCPGs that allow the locomotion of legged robots.…”

Section: Introductionmentioning

confidence: 99%

Evolutionary Spiking Neural Networks for Solving Supervised Classification Problems

López-Vázquez

Ornelas-Rodríguez

Espinal

et al. 2019

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

This paper presents a grammatical evolution (GE)-based methodology to automatically design third generation artificial neural networks (ANNs), also known as spiking neural networks (SNNs), for solving supervised classification problems. The proposal performs the SNN design by exploring the search space of three-layered feedforward topologies with configured synaptic connections (weights and delays) so that no explicit training is carried out. Besides, the designed SNNs have partial connections between input and hidden layers which may contribute to avoid redundancies and reduce the dimensionality of input feature vectors. The proposal was tested on several well-known benchmark datasets from the UCI repository and statistically compared against a similar design methodology for second generation ANNs and an adapted version of that methodology for SNNs; also, the results of the two methodologies and the proposed one were improved by changing the fitness function in the design process. The proposed methodology shows competitive and consistent results, and the statistical tests support the conclusion that the designs produced by the proposal perform better than those produced by other methodologies.

show abstract

Comparing Metaheuristic Algorithms on the Training Process of Spiking Neural Networks

Cited by 5 publications

References 9 publications

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control

Evolutionary Spiking Neural Networks for Solving Supervised Classification Problems

Contact Info

Product

Resources

About