Measuring what Really Matters: Optimizing Neural Networks for TinyML

Abstract

With the surge of inexpensive computational and memory resources, neural networks (NNs) have experienced an unprecedented growth in architectural and computational complexity. Introducing NNs to resource-constrained devices enables cost-efficient deployments, widespread availability, and the preservation of sensitive data.This work addresses the challenges of bringing Machine Learning to microcontroller units (MCUs), where we focus on the ubiquitous ARM Cortex-M architecture. The detailed effects and trade-offs that optimization methods, software frameworks, and MCU hardware architecture have on key performance metrics such as inference latency and energy consumption have not been previously studied in depth for state-of-the-art frameworks such as TensorFlow Lite Micro. We find that empirical investigations which measure the perceptible metrics -performance as experienced by the userare indispensable, as the impact of specialized instructions and layer types can be subtle. To this end, we propose an implementation-aware design as a cost-effective method for verification and benchmarking. Employing our developed toolchain, we demonstrate how existing NN deployments on resource-constrained devices can be improved by systematically optimizing NNs to their targeted application scenario.

Energy-Efficient Inference on the Edge Exploiting TinyML Capabilities for UAVs

“…Although quantization is still an open research topic, it has become a standard compression method for TinyMLrelated applications. It allows the consumption of less flash memory and RAM whilst maintaining almost the same accuracy of the original model [28]. Moreover, compressing the network from 32-bit to 8-bit results in a significantly faster processing, shorter inference time, and lower power consumption.…”

Achieving High Efficiency: Resource sharing techniques in artificial neural networks for resource-constrained devices

“…Table 1: Comparative overview of the digital hardware capabilities Software [26], [25] [27] MCUs [17], [18], [21] [22] FPGA [13], [14], [15] [19], [23] ASIC [14], [20 It can be seen from the table that while software, microcontrollers, FPGAs, and Application Specific Integrated Circuits (ASICs) can all implement ANNs, FPGAs offer distinct advantages in terms of performance, efficiency, flexibility, resource utilization, scalability, real-time capabilities, and price.…”

Machine Learning With Computer Networks: Techniques, Datasets, and Models

“…The core implementation of TinyML relies on ML model quantization, which reduces its numerical precision and size. Hence, implementing TinyML in environments with limited resources presents several ongoing challenges -The low computational capabilities and storage capacities of smaller devices restrict the complexity of the models that can be deployed [351]. This constraint can adversely affect the efficacy and precision of TinyML-based applications.…”