Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks

Tan, Weihao; Patel, Devdhar; Kozma, Róbert

doi:10.48550/arxiv.2009.14456

Cited by 3 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the limitation of resources, we choose 17 topperforming Atari games selected by [19] to test the method. The network architecture and hyper-parameters keep identical across all 17 games.…”

Section: Experimental Settingsmentioning

confidence: 99%

See 1 more Smart Citation

Deep Reinforcement Learning with Spiking Q-learning

Chen¹,

Peng²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combing SNNs and deep reinforcement learning (RL). There are only a few existing SNN-based RL methods at present. Most of them either lack generalization ability or employ Artificial Neural Networks (ANNs) to estimate value function in training. The former needs to tune numerous hyper-parameters for each scenario, and the latter limits the application of different types of RL algorithm and ignores the large energy consumption in training. To develop a robust spike-based RL method, we draw inspiration from non-spiking interneurons found in insects and propose the deep spiking Q-network (DSQN), using the membrane voltage of nonspiking neurons as the representation of Q-value, which can directly learn robust policies from highdimensional sensory inputs using end-to-end RL. Experiments conducted on 17 Atari games demonstrate the effectiveness of DSQN by outperforming the ANN-based deep Q-network (DQN) in most games. Moreover, the experimental results show superior learning stability and robustness to adversarial attacks of DSQN.

show abstract

Section: Experimental Settingsmentioning

confidence: 99%

“…The DSQN is compared with DQN and Converted SNN (ANN-SNN) [19], where DQN and ANN-SNN are re-run under the same experimental setting for a fair comparison. As shown in well on most games, as shown in Figure 6 (see the supplementary for the learning curves of all Atari games).…”

Section: The Performance Of Dsqns On Atari Gamesmentioning

confidence: 99%

Deep Reinforcement Learning with Spiking Q-learning

Chen¹,

Peng²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These approaches are typically based on reward-modulated local plasticity rules that perform well in simple control tasks, but commonly fail in complex robotic control tasks due to limited optimization capability. Some methods directly convert Deep Q-Networks (DQNs) [6] to SNNs and achieve competitive scores on Atari games with discrete action space [34,35].…”

Section: Related Workmentioning

confidence: 99%

Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning

Zhang¹,

Zhang²,

Jia³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the Deep Neural Networks (DNNs) as a powerful function approximator, Deep Reinforcement Learning (DRL) has been excellently demonstrated on robotic control tasks. Compared to DNNs with vanilla artificial neurons, the biologically plausible Spiking Neural Network (SNN) contains a diverse population of spiking neurons, making it naturally powerful on state representation with spatial and temporal information. Based on a hybrid learning framework, where a spike actor-network infers actions from states and a deep critic network evaluates the actor, we propose a Population-coding and Dynamic-neurons improved Spiking Actor Network (PDSAN) for efficient state representation from two different scales: input coding and neuronal coding. For input coding, we apply population coding with dynamically receptive fields to directly encode each input state component. For neuronal coding, we propose different types of dynamic-neurons (containing 1st-order and 2nd-order neuronal dynamics) to describe much more complex neuronal dynamics. Finally, the PDSAN is trained in conjunction with deep critic networks using the Twin Delayed Deep Deterministic policy gradient algorithm (TD3-PDSAN). Extensive experimental results show that our TD3-PDSAN model achieves better performance than state-of-the-art models on four OpenAI gym benchmark tasks. It is an im- * Duzhen Zhang and Tielin Zhang are co-first authors of this paper * *

show abstract

“…Because of the optimizing hardness and learning latency, agent based on SNN is challenging to be trained in reinforcement learning tasks. ANN-to-SNN conversion method (Rueckauer et al, 2017 ) is used to implement DQN with a spiking neural network (Patel et al, 2019 ; Tan et al, 2020 ). They first trained ANN-based DQN policy and then transferred the network weight to SNN, using SNN to plat Atari game as shown in Figure 1 .…”

Section: Introductionmentioning

confidence: 99%

Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization

Sun

Zeng

2022

Front. Neurosci.

View full text Add to dashboard Cite

Brain-inspired spiking neural networks (SNNs) are successfully applied to many pattern recognition domains. The SNNs-based deep structure has achieved considerable results in perceptual tasks, such as image classification and target detection. However, applying deep SNNs in reinforcement learning (RL) tasks is still a problem to be explored. Although there have been previous studies on the combination of SNNs and RL, most focus on robotic control problems with shallow networks or using the ANN-SNN conversion method to implement spiking deep Q networks (SDQN). In this study, we mathematically analyzed the problem of the disappearance of spiking signal features in SDQN and proposed a potential-based layer normalization (pbLN) method to train spiking deep Q networks directly. Experiment shows that compared with state-of-art ANN-SNN conversion method and other SDQN works, the proposed pbLN spiking deep Q networks (PL-SDQN) achieved better performance on Atari game tasks.

show abstract

Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks

Cited by 3 publications

References 28 publications

Deep Reinforcement Learning with Spiking Q-learning

Deep Reinforcement Learning with Spiking Q-learning

Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning

Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization

Contact Info

Product

Resources

About