7.7 A 0.2-to-3.6TOPS/W Programmable Convolutional Imager SoC with In-Sensor Current-Domain Ternary-Weighted MAC Operations for Feature Extraction and Region-of-Interest Detection

Lefebvre, Martin; Moreau, Ludovic; Dekimpe, Rémi; Bol, David

doi:10.1109/isscc42613.2021.9365839

Cited by 37 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these developments, the function of the device is fixed by the hardware circuit implemented in each pixel. Some configurability has been demonstrated, for instance by implementing programmable convolution kernels (26)(27)(28) or local binary patterns (29,30), but in general, although special-purpose vision sensors can be efficient, their application is limited due to the pre-defined functionality of the device.…”

Section: Vision Chips With Pixel Level Processorsmentioning

confidence: 99%

Sensor-level computer vision with pixel processor arrays for agile robots

et al. 2022

View full text Add to dashboard Cite

Vision processing for control of agile autonomous robots requires low-latency computation, within a limited power and space budget. This is challenging for conventional computing hardware. Parallel processor arrays (PPAs) are a new class of vision sensor devices that exploit advances in semiconductor technology, embedding a processor within each pixel of the image sensor array. Sensed pixel data are processed on the focal plane, and only a small amount of relevant information is transmitted out of the vision sensor. This tight integration of sensing, processing, and memory within a massively parallel computing architecture leads to an interesting trade-off between high performance, low latency, low power, low cost, and versatility in a machine vision system. Here, we review the history of image sensing and processing hardware from the perspective of in-pixel computing and outline the key features of a state-of-the-art smart camera system based on a PPA device, through the description of the SCAMP-5 system. We describe several robotic applications for agile ground and aerial vehicles, demonstrating PPA sensing functionalities including high-speed odometry, target tracking, obstacle detection, and avoidance. In the conclusions, we provide some insight and perspective on the future development of PPA devices, including their application and benefits within agile, robust, adaptable, and lightweight robotics.

show abstract

Section: Vision Chips With Pixel Level Processorsmentioning

confidence: 99%

Sensor-level computer vision with pixel processor arrays for agile robots

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Energy-efficiency and cameras. There has been recent work on designing more power-efficient camera sensors [8,35,54], mixed-signal vision integrated circuits [43,80] and proces-sors [63,70]. The general relation between power consumption versus resolution, color and SNR however still applies.…”

Section: Related Workmentioning

confidence: 99%

NeuriCam: Video Super-Resolution and Colorization Using Key Frames

Veluri¹,

Saffari²,

Pernu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present NeuriCam, a key-frame video super-resolution and colorization based system, to achieve low-power video capture from dual-mode IOT cameras. Our idea is to design a dual-mode camera system where the first mode is low power (1.1 mW) but only outputs gray-scale, low resolution and noisy video and the second mode consumes much higher power (100 mW) but outputs color and higher resolution images. To reduce total energy consumption, we heavily duty cycle the high power mode to output an image only once every second. The data from this camera system is then wirelessly streamed to a nearby plugged-in gateway, where we run our real-time neural network decoder to reconstruct a higher resolution color video. To achieve this, we introduce an attention feature filter mechanism that assigns different weights to different features, based on the correlation between the feature map and contents of the input frame at each spatial location. We design a wireless hardware prototype using off-the-shelf cameras and address practical issues including packet loss and perspective mismatch. Our evaluation shows that our dual-camera hardware reduces camera energy consumption while achieving an average gray-scale PSNR gain of 3.7 dB over prior video super resolution methods and 5.6 dB RGB gain over existing color propagation methods.Open-source code: https://github.com/vb000/NeuriCam

show abstract

“…TFT system integration: First, better datapath and peripherals for in situ CiM processing with raw sensor data require further design, which can refer to in-sensor computing designs based on the CMOS image sensor [4,112]. And particularly for TFT technology, reducing the use of high-cost TFTbased peripheral circuits can have much benefit.…”

Section: Future Of Tft-based Cim and Edge Computing Systemmentioning

confidence: 99%

Computing-in-memory with thin-filmtransistors: challenges and opportunities

Tang¹,

Liu²,

Li³

et al. 2022

Flex. Print. Electron.

View full text Add to dashboard Cite

Thin-film transistor (TFT) has attracted enormous interests recently for its great potential in a wide range of edge computing applications, benefitting from its large-area low-cost flexible fabrications, and well integration with sensors and displays. With the support of in-situ processing of sensor data, TFT-based edge systems show their advantages in large-scale dense sensing with real-time energy-efficient processing and interaction, and more excitingly, it provides the opportunity of eliminating the massive data transfer to the cloud servers. However, the design of high-performance processing modules based on TFT is difficult, due to large device variation, poor stability, and low mobility. Computing-in-memory (CiM), which has been proposed recently as a high-efficiency high-parallelism computing approach, is expected to improve the capacity of TFT-based edge computing systems. In this survey, various recent works on TFT-based CiM have been summarized, showing the superiority to conventional processing flow by efficient in-memory analog computation with mitigation of data transfer, and reduced ADC usage for sensor data. With both opportunities and challenges, the design space and trend of TFT-based CiM to be explored are then described. Finally, further development and co-optimization from device to system are discussed for flourishing of the next-generation intelligent TFT-based edge system.

show abstract

7.7 A 0.2-to-3.6TOPS/W Programmable Convolutional Imager SoC with In-Sensor Current-Domain Ternary-Weighted MAC Operations for Feature Extraction and Region-of-Interest Detection

Cited by 37 publications

References 5 publications

Sensor-level computer vision with pixel processor arrays for agile robots

Sensor-level computer vision with pixel processor arrays for agile robots

NeuriCam: Video Super-Resolution and Colorization Using Key Frames

Computing-in-memory with thin-filmtransistors: challenges and opportunities

Contact Info

Product

Resources

About