Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware

Abstract

Energy-efficient mapless navigation is crucial for mobile robots as they explore unknown environments with limited on-board resources. Although the recent deep reinforcement learning (DRL) approaches have been successfully applied to navigation, their high energy consumption limits their use in many robotic applications. Here, we propose a neuromorphic approach that combines the energy-efficiency of spiking neural networks with the optimality of DRL to learn control policies for mapless navigation. Our hybrid framework, Spiking deep deterministic policy gradient (SDDPG), consists of a spiking actor network (SAN) and a deep critic network, where the two networks were trained jointly using gradient descent. The trained SAN was deployed on Intel's Loihi neuromorphic processor. The co-learning enabled synergistic information exchange between the two networks, allowing them to overcome each other's limitations through a shared representation learning. When validated on both simulated and real-world complex environments, our method on Loihi not only consumed 75 times less energy per inference as compared to DDPG on Jetson TX2, but also had a higher rate of successfully navigating to the goal which ranged by 1% to 4.2%, depending on the forward-propagation timestep size. These results reinforce our ongoing effort to design braininspired algorithms for controlling autonomous robots with neuromorphic hardware.

Spiking Neural Networks and Their Applications: A Review

“…SDDPG [91] (2020) PyTroch/-Spiking deep deterministic policy gradient (SDDPG), which consists of a spiking actor network and a deep critic network that were trained jointly using gradient descent for energy-efficient mapless navigation.…”

“…In this work, reference [ 90 ] shows that SNN may robustly control an autonomous robot in mapping and exploring an unknown environment, while compensating for its own intrinsic hardware imperfections, such as partial or total loss of visual input. Reference [ 91 ] proposed a variant of deep deterministic policy gradient (DDPG), called spiking deep deterministic policy gradient (SDDPG), which consists of a spiking actor network and a deep critic network that were trained jointly using gradient descent for energy-efficient mapless navigation. This work explored an indirect SNN training approach based on the reward-modulated spike-timing-dependent plasticity (R-STDP) learning rule and supervised learning framework.…”

Bioinspired smooth neuromorphic control for robotic arms

“…The interaction between Loihi and the simulation environments is realized using the framework proposed in [60] to allow for real-time control of the robot arms (figure 4). For this, we use a Read channel controlled by the low-frequency x86 Loihi cores (SNIP) to read out the spiking activity of the motor neurons and translate it to joint movements.…”

Representation Learning for Event-based Visuomotor Policies

“…Event camera based perception was used in other applications as well, such as self-supervised learning of optical flow [28], steering prediction for self driving cars [3]. Spiking neural networks were also used to examine eventbased data [29,30,31,32,33,34].…”