VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Abstract

Deep learning (DL) models have achieved great success in many application domains. As such, many industrial companies such as Google and Facebook have acknowledged the importance of multitenant DL services. Although the multi-tenant service has been studied in conventional workloads, it is not been deeply studied on deep learning service, especially on general-purpose hardware.In this work, we systematically analyze the opportunities and challenges of providing multi-tenant deep learning services on the general-purpose CPU architecture from the aspects of scheduling granularity and code generation. We propose an adaptive granularity scheduling scheme to both guarantee resource usage efficiency and reduce the scheduling conflict rate. We also propose an adaptive compilation strategy, by which we can dynamically and intelligently pick a program with proper exclusive and shared resource usage to reduce overall interference-induced performance loss. Compared to the existing works, our design can serve more requests under the same QoS target in various scenarios (e.g., +71%, +62%, +45% for light, medium, and heavy workloads, respectively), and reduce the averaged query latency by 50%. CCS CONCEPTS• Computer systems organization → Neural networks; Cloud computing; • Computing methodologies → Concurrent algorithms.

GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching

“…The quadratic nature of attention mechanisms has led to a substantial surge in memory consumption for LLMs, thereby magnifying the significance of effective memory management [8,16,58]. Researchers have proposed various algorithmic optimizations aimed at curbing memory consumption, including quantization techniques [22, 24-26, 32, 46, 47, 81, 94], pruning strategies [19-21, 27, 39, 52, 64, 65, 80, 93], and KV-cache compression approaches [5,37], compilation [9,34,[90][91][92] and scheduling [11,23,50,51,54,85].…”

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

“…There are some prior research on optimizing the operator scheduling of DNN models to improve the quality of model service [13], [25], [26], [27]. REEF [25] apopts a parallel mechanism based on dynamic kernel padding to improve the overall throughput.…”

“…REEF [25] apopts a parallel mechanism based on dynamic kernel padding to improve the overall throughput. VELTAIR [27] proposed an adaptive operator-level compilation and scheduling to guarantee resource usage efficiency and reduce interference-induced performance loss for multi-tenant DNN services. PREMA [13] is a predictive multi-task scheduling algorithm for preemptible neural processing unit to meet high-throughput.…”