Intelligence Beyond the Edge

Abstract

Energy-harvesting technology provides a promising platform for future IoT applications. However, since communication is very expensive in these devices, applications will require inference "beyond the edge" to avoid wasting precious energy on pointless communication. We show that application performance is highly sensitive to inference accuracy. Unfortunately, accurate inference requires large amounts of computation and memory, and energy-harvesting systems are severely resource-constrained. Moreover, energy-harvesting systems operate intermittently, suffering frequent power failures that corrupt results and impede forward progress. This paper overcomes these challenges to present the first full-scale demonstration of DNN inference on an energyharvesting system. We design and implement SONIC, an intermittence-aware software system with specialized support for DNN inference. SONIC introduces loop continuation, a new technique that dramatically reduces the cost of guaranteeing correct intermittent execution for loop-heavy code like DNN inference. To build a complete system, we further present GENESIS, a tool that automatically compresses networks to optimally balance inference accuracy and energy, and TAILS, which exploits SIMD hardware available in some microcontrollers to improve energy efficiency. Both SONIC & TAILS guarantee correct intermittent execution without any hand-tuning or performance loss across different power systems. Across three neural networks on a commercially available microcontroller, SONIC & TAILS reduce inference energy by 6.9× and 12.2×, respectively, over the state-of-the-art. CCS Concepts • Computer systems organization → Embedded software.

Partitioning and Placement of Deep Neural Networks on Distributed Edge Devices to Maximize Inference Throughput

“…2.1.2 Edge Inference Runtimes. Intermittent edge inference [21] describes how to optimize edge inference for energy use on edge devices, but focuses on compression and pruning of model layers with a specialized inference runtime. Jupiter [20] orchestrates execution of a task on geographically distributed compute nodes based on a given task graph.…”

Automatically enforcing fresh and consistent inputs in intermittent systems

“…After a system-specific amount of energy accumulates, hardware activates the system to begin executing, quickly consuming the energy (green segments). The executing system may collect sensor inputs, run computations (e.g., machine learning to process sensor data [16,17,38]) on an ultra-low-power CPU or microcontroller, and log or transmit results via a wireless radio link.…”

REDRESS: Generating Compressed Models for Edge Inference Using Tsetlin Machines

“…To enable true wide-spread integration of AI sensors for low and ultra-low power edge inference there is a need for focused design effort towards delivering energy efficient and memory frugal implementations [5], [6], [7]. The prevailing approach to edge inference is based on Neural Network (NN) models [5], [8].…”

“…To enable true wide-spread integration of AI sensors for low and ultra-low power edge inference there is a need for focused design effort towards delivering energy efficient and memory frugal implementations [5], [6], [7]. The prevailing approach to edge inference is based on Neural Network (NN) models [5], [8]. However, for a given inference application, designers are forced to choose intelligent trade-offs through hardware-software co-design considerations as the deep neural network (DNN) models are resource hungry in terms of storage, runtime memory (RAM) and computation.…”