A Neuromorphic Approach for Tracking using Dynamic Neural Fields on a Programmable Vision-chip

Abstract

In artificial vision applications, such as tracking, a large amount of data captured by sensors is transferred to processors to extract information relevant for the task at hand. Smart vision sensors offer a means to reduce the computational burden of visual processing pipelines by placing more processing capabilities next to the sensor. In this work, we use a vision-chip in which a small processor with memory is located next to each photosensitive element. The architecture of this device is optimized to perform local operations. To perform a task like tracking, we implement a neuromorphic approach using a Dynamic Neural Field, which allows to segregate, memorize, and track objects. Our system, consisting of the vision-chip running the DNF, outputs only the activity that corresponds to the tracked objects. These outputs reduce the bandwidth needed to transfer information as well as further post-processing, since computation happens at the pixel level. ABSTRACTIn artificial vision applications, such as tracking, a large amount of data captured by sensors is transferred to processors to extract information relevant for the task at hand. Smart vision sensors offer a means to reduce the computational burden of visual processing pipelines by placing more processing capabilities next to the sensor. In this work, we use a vision-chip in which a small processor with memory is located next to each photosensitive element. The architecture of this device is optimized to perform local operations.To perform a task like tracking, we implement a neuromorphic approach using a Dynamic Neural Field, which allows to segregate, memorize, and track objects. Our system, consisting of the vision-chip running the DNF, outputs only the activity that corresponds to the tracked objects. These outputs reduce the bandwidth needed to transfer information as well as further post-processing, since computation happens at the pixel level.

Fast Object Tracking on a Many-Core Neural Network Chip

“…We select an RNN model proposed by Wu et al (Fung et al, 2008 , 2010 ; Wu et al, 2008 ) named continuous attractor neural network (CANN), which is a neuroscience-inspired model. In fact, similar models with self-sustaining neural dynamics, termed as dynamic neural fields (DNF), can also be found in Faubel and Schöner ( 2008 ), Spencer and Perone ( 2008 ), Martel and Sandamirskaya ( 2016 ) and Schöner and Spencer ( 2016 ) where the only difference is the format of inhibition function.…”

“…Object tracking is important for many applications including autonomous driving, unmanned aerial vehicle, intelligent monitoring, etc. The object tracking models used by prior work can be clustered into several categories: discriminative or generative models (Li et al, 2013 ; Wang N. et al, 2015 ), machine learning models (Grabner et al, 2008 ; Wang and Yeung, 2013 ; Hare et al, 2016 ), and dynamic neural models (Faubel and Schöner, 2008 ; Spencer and Perone, 2008 ; Wu et al, 2008 ; Martel and Sandamirskaya, 2016 ). The generative models leverage specific characteristics to represent the object, i.e., using representative methods such as the PCA (Ross et al, 2008 ; Wang et al, 2013 ) and sparse coding methods (Jia et al, 2012 ; Zhang T. et al, 2012 ), while the discriminative models separate the object from the backgrounds by training binary classifier (Kalal et al, 2012 ; Zhang K. et al, 2012 ).…”

“…To realize fast object tracking still remains as a challenge but important for applications such as motion posture capture in sports field (Chen et al, 2015 ; Pueo, 2016 ), cell imaging and movement analysis in biomedical field (Beier and Ibey, 2014 ), and some real-life scenarios (Galoogahi et al, 2017a ). Compared to above complex models, the recurrent neural networks (RNNs) with attractor dynamics (Faubel and Schöner, 2008 ; Spencer and Perone, 2008 ; Wu et al, 2008 ; Martel and Sandamirskaya, 2016 ) are more promising for fast tracking. They are capable of holding a continuous family of stationary states and form a continuous manifold wherein the dynamic behavior is neutrally stable, facilitating the smoothness of the object tracking.…”

“…These characteristics together with their bulky size and huge energy consumption make it difficult for the embedded deployment. Previous work (Martel and Sandamirskaya, 2016 ) implemented neural dynamics on dedicated vision chip for efficient tracking benefit from the near-data processing. Whereas, the flexibility of programming and application is limited due to the tight integration with the vision sensor and the effectiveness on various video datasets is yet to be fully demonstrated.…”

A Demonstration of Tracking using Dynamic Neural Fields on a Programmable Vision Chip

“…The tracking system demonstrated in this work is presented in greater detail in [2]. It comprises a programmable vision chip on which a DNF implementation runs over a saliency map.…”