Bio-inspired smart vision sensor: toward a reconfigurable hardware modeling of the hierarchical processing in the brain

Bhowmik, Pankaj; Pantho, Jubaer Hossain; Bobda, Christophe

doi:10.1007/s11554-020-00960-5

Cited by 15 publications

(12 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that detailed specifications of the curved neuromorphic imaging device are compared to those of the relevant neuromorphic image sensors in Supplementary Table 3. Nevertheless, additional processors for data post-processing, which extract features from the pre-processed image data and identify the target object, are still necessary for machine vision applications 47,48 . Therefore, further device units for efficient post-processing of the pre-processed image data should be still integrated 37 , although the pre-processed image can be efficiently obtained by cNISA.…”

Section: Discussionmentioning

confidence: 99%

Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system

et al. 2020

View full text Add to dashboard Cite

Conventional imaging and recognition systems require an extensive amount of data storage, pre-processing, and chip-to-chip communications as well as aberration-proof light focusing with multiple lenses for recognizing an object from massive optical inputs. This is because separate chips (i.e., flat image sensor array, memory device, and CPU) in conjunction with complicated optics should capture, store, and process massive image information independently. In contrast, human vision employs a highly efficient imaging and recognition process. Here, inspired by the human visual recognition system, we present a novel imaging device for efficient image acquisition and data pre-processing by conferring the neuromorphic data processing function on a curved image sensor array. The curved neuromorphic image sensor array is based on a heterostructure of MoS2 and poly(1,3,5-trimethyl-1,3,5-trivinyl cyclotrisiloxane). The curved neuromorphic image sensor array features photon-triggered synaptic plasticity owing to its quasi-linear time-dependent photocurrent generation and prolonged photocurrent decay, originated from charge trapping in the MoS2-organic vertical stack. The curved neuromorphic image sensor array integrated with a plano-convex lens derives a pre-processed image from a set of noisy optical inputs without redundant data storage, processing, and communications as well as without complex optics. The proposed imaging device can substantially improve efficiency of the image acquisition and recognition process, a step forward to the next generation machine vision.

show abstract

Section: Discussionmentioning

confidence: 99%

Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The retina performs some preprocessing to determine the regions where the visual attention implies. Inheriting these biological vision systems in vision applications provides the firmest connection to the low-dimensional fixational space and high-dimensional features or object space [ 10 , 11 ]. Hence, computationally expensive operations can be triggered by lightweight algorithms in the circuit to emulate vision systems [ 11 ].…”

Section: Introductionmentioning

confidence: 99%

“…Inheriting these biological vision systems in vision applications provides the firmest connection to the low-dimensional fixational space and high-dimensional features or object space [ 10 , 11 ]. Hence, computationally expensive operations can be triggered by lightweight algorithms in the circuit to emulate vision systems [ 11 ]. This concept is tailored in the proposed HARP architecture by dividing the image into small image patches and then investigating the relevance of the image patches or regions.…”

Section: Introductionmentioning

confidence: 99%

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Bhowmik

Pantho

Bobda

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.

show abstract

“…Based on the information-theoretic considerations, in the biological vision system, the role of early sensory processing is to reduce redundancy [ 30 ]. According to this model, the visual system uses an internal model to predict incoming signals and reduces redundancy by removing the repeated components [ 31 , 32 ].…”

Section: Introductionmentioning

confidence: 99%

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

Pantho

Bhowmik

Bobda

2021

Sensors

Self Cite

View full text Add to dashboard Cite

The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations’ overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors’ computational capabilities.

show abstract

Bio-inspired smart vision sensor: toward a reconfigurable hardware modeling of the hierarchical processing in the brain

Cited by 15 publications

References 43 publications

Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system

Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

Contact Info

Product

Resources

About