Visual Cortex Inspired Pixel-Level Re-configurable Processors for Smart Image Sensors

Bhowmik, Pankaj; Pantho, Jubaer Hossain; Bobda, Christophe

doi:10.1145/3316781.3322481

Cited by 6 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Systematic integration of computing and sensor arrays has been widely studied to eliminate off-chip data transmission and reduce ADC bandwidth by combining CMOS image sensor and processors in one chip as known as PNS [2], [13], [21]- [24], or even integrating pixels and computation unit so-called PIS [12], [17], [25]- [28]. In [13], photocurrents are transformed into pulse-width modulation signals and a dedicated analog processor is designed to execute feature extraction reducing ADC power consumption.…”

Section: A Near-sensor and In-sensor Processingmentioning

confidence: 99%

PISA: A Binary-Weight Processing-In-Sensor Accelerator for Edge Image Processing

Angizi¹,

Tabrizchi²,

Roohi³

2022

Preprint

View full text Add to dashboard Cite

This work proposes a Processing-In-Sensor Accelerator, namely PISA, as a flexible, energy-efficient, and highperformance solution for real-time and smart image processing in AI devices. PISA intrinsically implements a coarse-grained convolution operation in Binarized-Weight Neural Networks (BWNNs) leveraging a novel compute-pixel with non-volatile weight storage at the sensor side. This remarkably reduces the power consumption of data conversion and transmission to an off-chip processor. The design is completed with a bit-wise near-sensor processing-in-DRAM computing unit to process the remaining network layers. Once the object is detected, PISA switches to typical sensing mode to capture the image for a fine-grained convolution using only the near-sensor processing unit. Our circuit-to-application co-simulation results on a BWNN acceleration demonstrate acceptable accuracy on various image datasets in coarse-grained evaluation compared to baseline BWNN models, while PISA achieves a frame rate of 1000 and efficiency of ∼1.74 TOp/s/W. Lastly, PISA substantially reduces data conversion and transmission energy by ∼84% compared to a baseline CPU-sensor design.

show abstract

Section: A Near-sensor and In-sensor Processingmentioning

confidence: 99%

PISA: A Binary-Weight Processing-In-Sensor Accelerator for Edge Image Processing

Angizi¹,

Tabrizchi²,

Roohi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Systematic integration of computing and sensor arrays has been widely studied to eliminate off-chip data transmission and reduce ADC bandwidth by combining CMOS image sensors and processors in one chip, known as PNS [8][9][10][11][12], or even integrating pixels and computation unit so-called PIS [13][14][15][16]. In [9], photocurrents are transformed into pulse-width modulation signals, and a dedicated analog processor is designed to execute feature extraction reducing ADC power consumption.…”

Section: Near/in-sensor Processing Backgroundmentioning

confidence: 99%

Ocelli: Efficient Processing-in-Pixel Array Enabling Edge Inference of Ternary Neural Networks

Tabrizchi

Angizi

Roohi

2022

JLPEA

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs), due to their recent successes, have gained lots of attention in various vision-based applications. They have proven to produce incredible results, especially on big data, that require high processing demands. However, CNN processing demands have limited their usage in embedded edge devices with constrained energy budgets and hardware. This paper proposes an efficient new architecture, namely Ocelli includes a ternary compute pixel (TCP) consisting of a CMOS-based pixel and a compute add-on. The proposed Ocelli architecture offers several features; (I) Because of the compute add-on, TCPs can produce ternary values (i.e., −1, 0, +1) regarding the light intensity as pixels’ inputs; (II) Ocelli realizes analog convolutions enabling low-precision ternary weight neural networks. Since the first layer’s convolution operations are the performance bottleneck of accelerators, Ocelli mitigates the overhead of analog buffers and analog-to-digital converters. Moreover, our design supports a zero-skipping scheme to further power reduction; (III) Ocelli exploits non-volatile magnetic RAMs to store CNN’s weights, which remarkably reduces the static power consumption; and finally, (IV) Ocelli has two modes, including sensing and processing. Once the object is detected, the architecture switches to the typical sensing mode to capture the image. Compared to the conventional pixels, it achieves an average 10% efficiency on its lane detection power consumption compared with existing edge detection algorithms. Moreover, considering different CNN workloads, our design shows more than 23% power efficiency over conventional designs, while it can achieve better accuracy.

show abstract

“…Determining spatiotemporal saliency is one of the common approaches to identify the relevance of a region [ 13 ]. HARP utilizes some low-level image processing algorithms (edge, corner, temporal contrast filter, predictive coding [ 14 ]) instead of complicated calculations. Our approach’s difference is that HARP does not generate a saliency map; instead, it investigates a region’s possibility to be marked as relevant with a series of low-level image processing.…”

Section: Introductionmentioning

confidence: 99%

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Bhowmik

Pantho

Bobda

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.

show abstract

Visual Cortex Inspired Pixel-Level Re-configurable Processors for Smart Image Sensors

Cited by 6 publications

References 3 publications

PISA: A Binary-Weight Processing-In-Sensor Accelerator for Edge Image Processing

PISA: A Binary-Weight Processing-In-Sensor Accelerator for Edge Image Processing

Ocelli: Efficient Processing-in-Pixel Array Enabling Edge Inference of Ternary Neural Networks

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Contact Info

Product

Resources

About