In this paper we demonstrate how the Nengo neural modeling and simulation libraries enable users to quickly develop robotic perception and action neural networks for simulation on neuromorphic hardware using tools they are already familiar with, such as Keras and Python. We identify four primary challenges in building robust, embedded neurorobotic systems, including: (1) developing infrastructure for interfacing with the environment and sensors; (2) processing task specific sensory signals; (3) generating robust, explainable control signals; and (4) compiling neural networks to run on target hardware. Nengo helps to address these challenges by: (1) providing the NengoInterfaces library, which defines a simple but powerful API for users to interact with simulations and hardware; (2) providing the NengoDL library, which lets users use the Keras and TensorFlow API to develop Nengo models; (3) implementing the Neural Engineering Framework, which provides white-box methods for implementing known functions and circuits; and (4) providing multiple backend libraries, such as NengoLoihi, that enable users to compile the same model to different hardware. We present two examples using Nengo to develop neural networks that run on CPUs and GPUs as well as Intel's neuromorphic chip, Loihi, to demonstrate two variations on this workflow. The first example is an implementation of an end-to-end spiking neural network in Nengo that controls a rover simulated in Mujoco. The network integrates a deep convolutional network that processes visual input from cameras mounted on the rover to track a target, and a control system implementing steering and drive functions in connection weights to guide the rover to the target. The second example uses Nengo as a smaller component in a system that has addressed some but not all of those challenges. Specifically it is used to augment a force-based operational space controller with neural adaptive control to improve performance during a reaching task using a real-world Kinova Jaco 2 robotic arm. The code and implementation details are provided 1 , with the intent of enabling other researchers to build and run their own neurorobotic systems.

No abstract

No abstract

In this paper, we present a fully spiking neural network running on Intel’s Loihi chip for operational space control of a simulated 7-DOF arm. Our approach uniquely combines neural engineering and deep learning methods to successfully implement position and orientation control of the end effector. The development process involved 4 stages: 1) Designing a node-based network architecture implementing an analytical solution; 2) developing rate neuron networks to replace the nodes; 3) retraining the network to handle spiking neurons and temporal dynamics; and finally 4) adapting the network for the specific hardware constraints of the Loihi. We benchmark the controller on a center-out reaching task, using the deviation of the end effector from the ideal trajectory as our evaluation metric. The RMSE of the final neuromorphic controller running on Loihi is only slightly worse than the analytic solution, with 4.13% more deviation from the ideal trajectory, and uses two orders of magnitude less energy per inference than standard hardware solutions. While qualitative discrepancies remain, we find these results support both our approach and the potential of neuromorphic controllers. To the best of our knowledge, this work represents the most advanced neuromorphic implementation of neurorobotics developed to date.

Rapid developments in technologies such as embedded devices, with increased processing capability, and sensor systems, with high accuracy and fast response, are enabling a wide range of new road traffic applications. These new systems and services promise significant improvements in areas such road safety, fuel economy and congestion management. In many cases the commercial deployment of these Advanced Driver Assistance Systems (ADAS), or Intelligent Transportation Systems (ITS), is no longer limited by just technical or economic constraints. There are ethical, legal or certification issues that need to be resolved for wide scale deployment.In this paper we present the Network Assisted Vehicle (NAV) concept. NAV is a prototype semi-autonomous vehicle with a modular design that can be adapted to support new ADAS and ITS test standards. Vehicle costs are minimized by making full use of existing OEM systems on the vehicle and the known conditions of a controlled test environment (innovITS ADVANCE City Circuit test facility).