In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing

Abstract

As machine vision technology generates large amounts of data from sensors, it requires efficient computational systems for visual cognitive processing. Recently, in-sensor computing systems have emerged as a potential solution for reducing unnecessary data transfer and realizing fast and energy-efficient visual cognitive processing. However, they still lack the capability to process stored images directly within the sensor. Here, we demonstrate a heterogeneously integrated 1-photodiode and 1 memristor (1P-1R) crossbar for in-sensor visual cognitive processing, emulating a mammalian image encoding process to extract features from the input images. Unlike other neuromorphic vision processes, the trained weight values are applied as an input voltage to the image-saved crossbar array instead of storing the weight value in the memristors, realizing the in-sensor computing paradigm. We believe the heterogeneously integrated in-sensor computing platform provides an advanced architecture for real-time and data-intensive machine-vision applications via bio-stimulus domain reduction.

Nanomaterial-Based Synaptic Optoelectronic Devices for In-Sensor Preprocessing of Image Data

“…Classification by Synaptic Optoelectronic Devices. The roles of synaptic optoelectronic devices have been expanding to image filtering 5,14,127 and pattern classification, 8,16 which were originally achieved using conventional microprocessors with appropriate software. Synaptic optoelectronic devices can serve as image filters and ANNs, which can be used for feature extraction and image recognition.…”

“…Therefore, this front-end part of machine vision can be inefficient . In contrast, synaptic optoelectronic devices perform image preprocessing at the image-sensor level . Through in-sensor preprocessing performed by the image-sensing device itself, the preprocessed data can be obtained from massive raw image data without computationally expensive preprocessing steps .…”

“…The roles of synaptic optoelectronic devices have been expanding to image filtering ,, and pattern classification, , which were originally achieved using conventional microprocessors with appropriate software. Synaptic optoelectronic devices can serve as image filters and ANNs, which can be used for feature extraction and image recognition. , For example, Jang et al reported a 32 × 32 MoS 2 phototransistor array (Figure a), which captures incoming images as image sensors and also functions as a matrix multiplication engine for image filtering and recognition .…”

“…14 This is because the considerable amount of image data in the entire time domain should be acquired by the image sensor and then transferred to the processor for image data processing and recognition, consequently consuming significant energy and processing time. 15,16 One promising solution for this inefficiency that originated from the system configuration is to adopt a novel imagesensing and data-processing platform that emulates the human vision and image recognition system. 4,17,18 The human retina performs first-stage image preprocessing by the bipolar and ganglion cells as well as performs image acquisition by the photoreceptor cells.…”

“…Recent advances in image acquisition − and data processing , technologies have enabled diverse machine vision applications including facile object detection − and accurate image recognition, , leading to a new era in the development of surveillance, self-driving vehicles, and autonomous robotics technologies. , Simultaneously, the amount of image data to be acquired and processed by the image sensing and processing device for realizing the machine vision application has been exponentially growing, significantly increasing the computational burden. , However, the conventional architecture of the machine vision system, where the front-end image sensor and the back-end processor are physically separated, involves inefficiency in terms of power consumption and data latency . This is because the considerable amount of image data in the entire time domain should be acquired by the image sensor and then transferred to the processor for image data processing and recognition, consequently consuming significant energy and processing time. , …”