Steering a predator robot using a mixed frame/event-driven convolutional neural network

Moeys, Diederik Paul; Corradi, Federico; Kerr, Emmett; Vance, Philip; Das, Gautham; Neil, Daniel; Kerr, Dermot; Delbrück, Tobi

doi:10.1109/ebccsp.2016.7605233

Cited by 96 publications

(71 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Selecting the appropriate input representation of a set of events for a neural network is still a challenging problem. Prior works such as Moeys et al [14] and Maqueda et al [11] generate an event image by summing the number of events at each pixel. However, this discards the rich temporal information in the events, and is susceptible to motion blur.…”

Section: Input: the Discretized Event Volumementioning

confidence: 99%

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Zhu

Yuan

Chaney

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

314

292

View full text Add to dashboard Cite

In this work, we propose a novel framework for unsupervised learning for event cameras that learns motion information from only the event stream. In particular, we propose an input representation of the events in the form of a discretized volume that maintains the temporal distribution of the events, which we pass through a neural network to predict the motion of the events. This motion is used to attempt to remove any motion blur in the event image. We then propose a loss function applied to the motion compensated event image that measures the motion blur in this image. We train two networks with this framework, one to predict optical flow, and one to predict egomotion and depths, and evaluate these networks on the Multi Vehicle Stereo Event Camera dataset, along with qualitative results from a variety of different scenes.

show abstract

Section: Input: the Discretized Event Volumementioning

confidence: 99%

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Zhu

Yuan

Chaney

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

314

292

View full text Add to dashboard Cite

show abstract

“…RoshamboNet is a 5-layer, 20 MOp, 114k weight CNN architecture, described in Table V, trained to play the rockscissors-paper game [31]. This network can classify input images of size 64x64 obtained from the DVS of a DAVIS camera using the same training and feature extraction stage approach from [32]. The network outputs 4 classes: "rock", "scissors", "paper" or "background" from each feature vector.…”

Section: Roshambonetmentioning

confidence: 99%

“…The face detector is a small CNN designed to recognize whether a face is present or absent in an image obtained from the DAVIS camera. The DVS events are accumulated into 36x36 input images, again using the method of [32]. The network was trained on a dataset of 1800k frames collected from public face datasets and labeled DAVIS frames.…”

Section: Face Detector Cnnmentioning

confidence: 99%

NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps

Aimar

Mostafa

Calabrese

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

235

177

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving many state-of-the-art (SOA) visual processing tasks. Even though graphical processing units are most often used in training and deploying CNNs, their power efficiency is less than 10 GOp/s/W for single-frame runtime inference. We propose a flexible and efficient CNN accelerator architecture called NullHop that implements SOA CNNs useful for low-power and low-latency application scenarios. NullHop exploits the sparsity of neuron activations in CNNs to accelerate the computation and reduce memory requirements. The flexible architecture allows high utilization of available computing resources across kernel sizes ranging from 1x1 to 7x7. NullHop can process up to 128 input and 128 output feature maps per layer in a single pass. We implemented the proposed architecture on a Xilinx Zynq field-programmable gate array (FPGA) platform and presented the results showing how our implementation reduces external memory transfers and compute time in five different CNNs ranging from small ones up to the widely known large VGG16 and VGG19 CNNs. Postsynthesis simulations using Mentor Modelsim in a 28-nm process with a clock frequency of 500 MHz show that the VGG19 network achieves over 450 GOp/s. By exploiting sparsity, NullHop achieves an efficiency of 368%, maintains over 98% utilization of the multiply-accumulate units, and achieves a power efficiency of over 3 TOp/s/W in a core area of 6.3 mm₂. As further proof of NullHop's usability, we interfaced its FPGA implementation with a neuromorphic event camera for real-time interactive demonstrations.

show abstract

“…Hence, we argue that researchers in the field should select a dataset onto which SNN accelerators could be compared fairly, where timing information is relevant, and no input conversion is required. Several event-driven datasets obtained with bioinspired image sensors have already been proposed [18,125,169,204].…”

Section: Low Power Spiking Machine Learningmentioning

confidence: 99%

Spiking Neural Networks Hardware Implementations and Challenges

Bouvier

Valentian

Mesquida

et al. 2019

J. Emerg. Technol. Comput. Syst.

179

View full text Add to dashboard Cite

Neuromorphic computing is henceforth a major research field for both academic and industrial actors. As opposed to Von Neumann machines, brain-inspired processors aim at bringing closer the memory and the computational elements to efficiently evaluate machine-learning algorithms. Recently, Spiking Neural Networks, a generation of cognitive algorithms employing computational primitives mimicking neuron and synapse operational principles, have become an important part of deep learning. They are expected to improve the computational performance and efficiency of neural networks, but are best suited for hardware able to support their temporal dynamics. In this survey, we present the state of the art of hardware implementations of spiking neural networks and the current trends in algorithm elaboration from model selection to training mechanisms. The scope of existing solutions is extensive; we thus present the general framework and study on a case-by-case basis the relevant particularities. We describe the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level and discuss their related advantages and challenges.

show abstract

Steering a predator robot using a mixed frame/event-driven convolutional neural network

Cited by 96 publications

References 7 publications

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps

Spiking Neural Networks Hardware Implementations and Challenges

Contact Info

Product

Resources

About