Scheduling Real-time Deep Learning Services as Imprecise Computations

Abstract

The paper presents an efficient real-time scheduling algorithm for intelligent real-time edge services, defined as those that perform machine intelligence tasks, such as voice recognition, LIDAR processing, or machine vision, on behalf of local embedded devices that are themselves unable to support extensive computations. The work contributes to a recent direction in real-time computing that develops scheduling algorithms for machine intelligence tasks with anytime predicition. We show that deep neural network workflows can be cast as imprecise computations, each with a mandatory part and (several) optional parts whose execution utility depends on input data. The goal of the real-time scheduler is to maximize average accuracy of deep neural network outputs, while meeting task deadlines, thanks to opportunistic shedding of the least necessary optional parts. The work is motivated by the proliferation of increasingly ubiquitous but resource-constrained embedded devices (for applications ranging from autonomous cars to the Internet of Things) and the desire to develop services that endow them with intelligence. Experiments on recent GPU hardware and a state of the art deep neural network for machine vision illustrate that our scheme can increase the overall accuracy by 10% ∼ 20%, while incurring (nearly) no deadline misses.

Adaptive Deep Learning for Soft Real-Time Image Classification

“…In fact, Refs. [24,25,49] are most closely related to our work. Our work is, however, significantly more comprehensive than them in that the number of layers in a or the number of resblocks in a ResNet (CNN) is only one hyper-parameter whose applicability is relatively limited as thoroughly analyzed in Section 4.…”

“…At run-time, it is possible to change paths based on deadlines. In [49], a ResNet [50], is divided into a mandatory part and an optional part where the former is always executed, but the latter is run when enough time is available. In fact, Refs.…”

IoT-Orchestration based Nanogrid Energy Management System and Optimal Time-Aware Scheduling for Efficient Energy Usage in Nanogrid

“…The IoT task execution follows a workflow with an end-to-end deadline in which an output of one task serves as an input for the followed task. In such a system, it is preferred for a job to produce approximate results than an overdue precise result [25]. Multiple studies have emphasized improving the orchestration and scheduling of such workloads in fog computing frameworks [22].…”

Improving security of autonomous cyber-physical systems against adversarial examples

“…The paper [104] schedules the neural network perception of partially selected regions within each input frame to meet the processing deadline on the GPU. The model-level scheduling adjusts the structure of deep models to meet the task deadlines [105].…”

“…Different from the above works [103][104][105][106], Sardino designs an application-specific planner to schedule the ensemble size at run time to meet a soft deadline.…”