TOD: Transprecise Object Detection to Maximise Real-Time Accuracy on the Edge

Abstract

Real-time video analytics on the edge is challenging as the computationally constrained resources typically cannot analyse video streams at full fidelity and frame rate, which results in loss of accuracy. This paper proposes a Transprecise Object Detector (TOD) which maximises the real-time object detection accuracy on an edge device by selecting an appropriate Deep Neural Network (DNN) on the fly with negligible computational overhead. TOD makes two key contributions over the state of the art: (1) TOD leverages characteristics of the video stream such as object size and speed of movement to identify networks with high prediction accuracy for the current frames; (2) it selects the best-performing network based on projected accuracy and computational demand using an effective and low-overhead decision mechanism. Experimental evaluation on a Jetson Nano demonstrates that TOD improves the average object detection precision by 34.7% over the YOLOv4-tiny-288 model on average over the MOT17Det dataset. In the MOT17-05 test dataset, TOD utilises only 45.1% of GPU resource and 62.7% of the GPU board power without losing accuracy, compared to YOLOv4-416 model. We expect that TOD will maximise the application of edge devices to real-time object detection, since TOD maximises real-time object detection accuracy given edge devices according to dynamic input features without increasing inference latency in practice.

Increased Leverage of Transprecision Computing for Machine Vision Applications at the Edge

“…There has been a considerable body of work on exploring runtime adaptation of NNs according to given compute resource constraints [1,[25][26][27][28][29]. The single NestDNN NN model [1] switches between multiple capacities of the NN during runtime according to accuracy and inference latency requirements.…”

“…Similarly, Yu et al [29] proposes the Slimmable Neural Network, in which the filter parameters are shared from a smaller capacity model to increase the capacity of the NN. Another study [25] proposes to use a runtime decision mechanism to switch between multiple NNs dynamically, according to video content and computational latency, in order to improve the real-time object detection accuracy.…”

Energy-aware Adaptive Approximate Computing for Deep Learning Applications

“…As shown in [8], on average about 83% of runtime computations for many machine learning and deep learning applications can be approximated. This can lead to substantial savings in power consumption, for instance in [9], the authors saved 63% of the power consumption by using approximations. This adaptive approximation can be performed on the hardware, e.g., using Dynamic Partial Reconfiguration (DPR) and by instantiating of different arithmetic hardware units [10], or using dynamic change of the frequency of operations [11].…”

“…This adaptive approximation can be performed on the hardware, e.g., using Dynamic Partial Reconfiguration (DPR) and by instantiating of different arithmetic hardware units [10], or using dynamic change of the frequency of operations [11]. It could also happen at the software level, e.g., by changing the utilized machine learning algorithm based on input data, such as [9] and [12], or by changing the memory access policy used in [13]. The benefits of these approaches can be up to 2.5× savings in the energy consumption of the system.…”

Toward Sustainable Serverless Computing

“…Consider the example of a real-time video analytics application, such as identifying objects on different frames of a video stream. A different DNN model from a portfolio of models can be employed for each frame to maximise the accuracy of detection [LVWV21]. This is achieved by leveraging the meta-characteristics of each video frame, such as the size of the object and the speed of movement of the object.…”