Anatomy of Deep Learning Image Classification and Object Detection on Commercial Edge Devices: A Case Study on Face Mask Detection

Abstract

Developing efficient on-the-edge Deep Learning (DL) applications is a challenging and non-trivial task, as first different DL models need to be explored with different trade-offs between accuracy and complexity, second, various optimization options, frameworks and libraries are available that need to be explored, third, a wide range of edge devices are available with different computation and memory constraints. As such, trade-offs arise among inference time, energy consumption, efficiency (throughput/watt) and value (throughput/dollar). To shed some light in this problem, a case study is delivered where seven Image Classification (IC) and six Object Detection (OD) State-of-The-Art (SOTA) DL models were used to detect face masks on the following commercial off-the-shelf edge devices: Raspberry PI 4, Intel Neural Compute Stick 2, Jetson Nano, Jetson Xavier NX, and i.MX 8M Plus. First, a full end-toend video pipeline face mask wearing detection architecture is developed. Then, the thirteen DL models were optimized, evaluated and compared on the edge devices, in terms of accuracy and inference time. To leverage the computational power of the edge devices, the models have been optimized, first, by using the SOTA optimization frameworks (TensorFlow Lite, OpenVINO, TensorRT, eIQ) and, second, by evaluating/comparing different optimization options, e.g., different levels of quantization. Note that the five edge devices are evaluated and compared too, in terms of inference time, value and efficiency. Last, we obtain insightful observations on which optimization frameworks, libraries and options to use and on how to select the right device depending on the target metric (inference time, efficiency and value). For example, we show that Jetson Xavier NX platform is the best in terms of latency and efficiency (FPS/Watt), while Jetson Nano is the best in terms of value (FPS/$).

Efficient Real-Time Smart Keyword Spotting Using Spectrogram-Based Hybrid CNN-LSTM for Edge System

“…In [40], a Hardware-Software codesign approach is used to make a DNN hardware accelerator, reaching 181ms inference time. For Jetson devices such as Jetson Nano, Jetson Xavier NX, and Jetson Xavier AGX, Jetson Xavier NX has the best performance in terms of FPS/Watt, Jetson Nano has the best performance in terms of FPS/$ [41], and Jetson Xavier AGX has the best performance in terms of calculated FPS [42].…”

Contactless Camera-Based Heart Rate and Respiratory Rate Monitoring Using AI on Hardware

“…Consequently, in the context of developing systems targeted for the edge, it is imperative to comprehend the trade-offs between the quality of the models, computational costs, commercial costs, and power consumption. In [35,36], several computer vision models are evaluated and compared to six popular edge devices in terms of accuracy and inference time.…”

Microbiological Quality Estimation of Meat Using Deep CNNs on Embedded Hardware Systems

“…Moreover, the machine learning modelling and estimation analysis are running offline on a computer, except for [19] presenting a handheld spectroscopic device with onboard statistical data analysis, but not machine learning-based analysis. The recent development of powerful microprocessors and embedded systems has allowed the development of solutions which run the artificial intelligence algorithms on the edge device, with the most popular applications of 'AI on edge' being in computer vision [20,21].…”