Nengo and Low-Power AI Hardware for Robust, Embedded Neurorobotics

Abstract

In this paper we demonstrate how the Nengo neural modeling and simulation libraries enable users to quickly develop robotic perception and action neural networks for simulation on neuromorphic hardware using tools they are already familiar with, such as Keras and Python. We identify four primary challenges in building robust, embedded neurorobotic systems, including: (1) developing infrastructure for interfacing with the environment and sensors; (2) processing task specific sensory signals; (3) generating robust, explainable control signals; and (4) compiling neural networks to run on target hardware. Nengo helps to address these challenges by: (1) providing the NengoInterfaces library, which defines a simple but powerful API for users to interact with simulations and hardware; (2) providing the NengoDL library, which lets users use the Keras and TensorFlow API to develop Nengo models; (3) implementing the Neural Engineering Framework, which provides white-box methods for implementing known functions and circuits; and (4) providing multiple backend libraries, such as NengoLoihi, that enable users to compile the same model to different hardware. We present two examples using Nengo to develop neural networks that run on CPUs and GPUs as well as Intel's neuromorphic chip, Loihi, to demonstrate two variations on this workflow. The first example is an implementation of an end-to-end spiking neural network in Nengo that controls a rover simulated in Mujoco. The network integrates a deep convolutional network that processes visual input from cameras mounted on the rover to track a target, and a control system implementing steering and drive functions in connection weights to guide the rover to the target. The second example uses Nengo as a smaller component in a system that has addressed some but not all of those challenges. Specifically it is used to augment a force-based operational space controller with neural adaptive control to improve performance during a reaching task using a real-world Kinova Jaco 2 robotic arm. The code and implementation details are provided 1 , with the intent of enabling other researchers to build and run their own neurorobotic systems.

Neuromorphic NEF-Based Inverse Kinematics and PID Control

“…NEF has the advantage of being able to deploy on numerous neuromorphic hardware. IK with NEF was demonstrated by DeWolf et al in their work on the REACH adaptive controller (DeWolf et al, 2016 ) and, more recently, in DeWolf et al ( 2020 ). REACH uses adaptive signals computed online (using PES-learning) to modulate arm movement to adapt to unexpected conditions.…”

“…As a result, the system converges to its target, as is evident from the decreasing error ( Figure 2B ) (Gosmann and Eliasmith, 2016 ). This modification of the intercept distribution is crucial for accurate representation in 5D space and it is briefly described in DeWolf et al ( 2020 ) and Gosmann and Eliasmith ( 2016 ). Our model relies on PES-based optimization, and it is therefore constrained to a prespecified learning rate.…”

“…One theoretical framework, which allows for efficient data encoding and decoding with spiking neurons is the Neural Engineering Framework (NEF) (Eliasmith and Anderson, 2003 ). NEF is one of the most utilized theoretical frameworks in neuromorphic computing, and it was used to design neuromorphic systems capable of perception, memory, and motor control (DeWolf et al, 2020 ). It serves as the foundation for Nengo, a Python-based “neural compiler,” which translates high-level descriptions to low-level neural models (Bekolay et al, 2014 ).…”

Neuromorphic Analog Implementation of Neural Engineering Framework-Inspired Spiking Neuron for High-Dimensional Representation

“…This is a clear example of the importance of being able to modulate neuron’s tuning curves in high-dimensional representation. The importance of the discussion earlier was recently highlighted in DeWolf et al (2020) in the context of neuro-robotics.…”

“…NEF is one of the most utilized theoretical frameworks in neuromorphic computing. It was adopted for various neuromorphic tasks, ranging from neuro-robotics ( DeWolf et al, 2020 ) to high-level cognition ( Eliasmith et al, 2012 ). It was compiled to work on multiple neuromorphic hardware using Nengo, a Python-based “neural compiler,” which translates high-level descriptions to low-level neural models ( Bekolay et al, 2014 ).…”