The Improved Algorithm of Deep Q-learning Network Based on Eligibility Trace

Liu, Bingyan; Ye, Xiongbing; Zhou, Chifei; Liu, Yijing; Zhang, Qiyang; Dong, Fang

doi:10.1109/iccar49639.2020.9108040

Cited by 2 publications

(1 citation statement)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared with the backpropagation on spike-train level, the local synaptic plasticity is expected to apply on RSNNs, which consumes fewer computations and memories (Larsen and Sjöström, 2015;Kaiser et al, 2020). Bellec et al (2019Bellec et al ( , 2020 propose the eligibility backpropagation (e-prop) algorithm to replace unfolding the RSNN through time by the surrogate-gradient based on eligibility traces, which is known as the fading memory of events (Liu et al, 2020;Kalhor et al, 2021). Benefit from the local learning in synapses, the e-prop algorithm is considered suitable for mapping to the circuits like field programmable gate array (FPGA).…”

Section: Introductionmentioning

confidence: 99%

Presynaptic spike-driven plasticity based on eligibility trace for on-chip learning system

Gao

Deng²,

Wang³

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

IntroductionRecurrent spiking neural network (RSNN) performs excellently in spatio-temporal learning with backpropagation through time (BPTT) algorithm. But the requirement of computation and memory in BPTT makes it hard to realize an on-chip learning system based on RSNN. In this paper, we aim to realize a high-efficient RSNN learning system on field programmable gate array (FPGA).MethodsA presynaptic spike-driven plasticity architecture based on eligibility trace is implemented to reduce the resource consumption. The RSNN with leaky integrate-and-fire (LIF) and adaptive LIF (ALIF) models is implemented on FPGA based on presynaptic spike-driven architecture. In this architecture, the eligibility trace gated by a learning signal is used to optimize synaptic weights without unfolding the network through time. When a presynaptic spike occurs, the eligibility trace is calculated based on its latest timestamp and drives synapses to update their weights. Only the latest timestamps of presynaptic spikes are required to be stored in buffers to calculate eligibility traces.ResultsWe show the implementation of this architecture on FPGA and test it with two experiments. With the presynaptic spike-driven architecture, the resource consumptions, including look-up tables (LUTs) and registers, and dynamic power consumption of synaptic modules in the on-chip learning system are greatly reduced. The experiment results and compilation results show that the buffer size of the on-chip learning system is reduced and the RSNNs implemented on FPGA exhibit high efficiency in resources and energy while accurately solving tasks.DiscussionThis study provides a solution to the problem of data congestion in the buffer of large-scale learning systems.

show abstract