FA2: Fast, Accurate Autoscaling for Serving Deep Learning Inference with SLA Guarantees

Abstract

Deep learning (DL) inference has become an essential building block in modern intelligent applications. Due to the high computational intensity of DL, it is critical to scale DL inference serving systems in response to fluctuating workloads to achieve resource efficiency. Meanwhile, intelligent applications often require strict service level agreements (SLAs), which need to be guaranteed when the system is scaled. The problem is complex and has been tackled only in simple scenarios so far.This paper describes FA2, a fast and accurate autoscaler concept for DL inference serving systems. In contrast to related works, FA2 adopts a general, contrived two-phase approach. Specifically, it starts by capturing the autoscaling challenges in a comprehensive graph-based model. Then, FA2 applies targeted graph transformation and makes autoscaling decisions with an efficient algorithm based on dynamic programming. We implemented FA2 and built and evaluated a prototype. Compared with state-of-the-art autoscaling solutions, our experiments showed FA2 to achieve significant resource reduction (19% under CPUs and 25% under GPUs, on average) in combination with low SLA violations (less than 1.5%). FA2 performed close to the theoretical optimum, matching exactly the optimal decisions (with the least required resources) in 96.8% of all the cases in our evaluation.

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

“…Moreover, high accuracy is crucial for these services [20,26]. Consequently, inference systems must deliver highly accurate predictions with fewer computing resources (cost-efficient) while meeting latency constraints under workload variations [18,19,29,37].…”

IRIS: Interference and Resource Aware Predictive Orchestration for ML Inference Serving

“…Towards this direction, several works with different approaches have been conducted, aiming to address the problem of efficient resource orchestration and optimization for ML inference serving systems. Adaptive and pre-defined batching techniques [30]- [33] have been introduced aiming to support ML inference, while auto-scaling approaches are also considered [32], [34], [35], [35], [36]. Furthermore, aiming to support efficient ML inference, serverless approaches have been considered [33], [37], while MLbased and predictive solutions for load request and resource utilization have been widely utilized [13], [14], [38]- [43], [43], [44], such as reinforcement learning-based solutions [43].…”

“…Figure 9 indicates the QoS violations(top) and average CPU utilization(middle) for the image classification task. For low (20) and medium (35) target QoS constraint, the modelless scheduler mostly uses the ONNX-Mobilenet benchmark, which never violates the constraint. With the high QoS constraint (50), most of the time the TF-Mobilenet benchmark is used, which has a higher QPS capability with a lower CPU utilization.…”

Jellyfish: Timely Inference Serving for Dynamic Edge Networks

“…Clockwork [11] leverages the predictable performance of the DNNs, considers the SLO guarantees on the server, and maps requests to the desired model, but does not utilize DNN adaptation. Inferline [44], Llama [45], and FA2 [46] optimize the serving of complex DNN pipelines. INFaas [12] automates the hardware and model-variant selection and deployment through managed services.…”