Machine Learning In-Sensors: Computation-enabled Intelligent Sensors For Next Generation of IoT

Abstract

Smart sensors are an emerging technology that allows combining the data acquisition with the elaboration directly on the Edge device, very close to the sensors. To push this concept to the extreme, technology companies are proposing a new generation of sensors allowing to move the intelligence from the edge host device, typically a microcontroller, directly to the ultra-low-power sensor itself, in order to further reduce the miniaturization, cost and energy efficiency. This paper evaluates the capabilities of a novel and promising solution from STMicroelectronics. The presence of a floating point unit and an accelerator for binary neural networks provide capabilities for in-sensor feature extraction and machine learning. We propose a comparison of full-precision and binary neural networks for activity recognition with accelerometer data generated by the sensor itself. Experimental results have demonstrated that the sensor can achieve an inference performance of 10.7 cycles/MAC, comparable to a Cortex-M4-based microcontroller, with fullprecision networks, and up to 1.5 cycles/MAC with large binary models for low latency inference, with an average energy consumption of only 90 µJ/inference with the core running at 5 MHz.

TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation

“…Emerging platforms like Sony IMX500 [12] embed AI in sensors, enabling real-time processing at the data source, reducing latency, enhancing privacy by eliminating data transmission. To leverage the power of this platforms, algorithm optimization, model compression, quantization are vital [13]- [19] and redesigning AI models for effective operation within memory, computational constraints becomes crucial.…”