Enable simultaneous DNN services based on deterministic operator overlap and precise latency prediction

Cui, Weihao; Zhao, Han; Chen, Quan; Zheng, Ningxin; Leng, Jingwen; Zhao, Jieru; Song, Zhuo; Ma, Tao; Yang, Yong; Li, Chao; Guo, Minyi

doi:10.1145/3458817.3476143

Cited by 38 publications

(17 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ebird [13] also proposes a batchbased approach to enable concurrent execution of DNNs with high data transfer-compute overlapping. Abacus [14] proposes an operator overlapping strategy based on precise latency prediction. Besides the multi-DNN serving scenario, emerging microservicebased workloads also have complex inner structures similar to DNN models [20], to which our design may also be applied.…”

Section: Multi-tenant Deep Learning Servicementioning

confidence: 99%

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Liu

Leng

Zhang

et al. 2022

Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

Self Cite

View full text Add to dashboard Cite

Deep learning (DL) models have achieved great success in many application domains. As such, many industrial companies such as Google and Facebook have acknowledged the importance of multitenant DL services. Although the multi-tenant service has been studied in conventional workloads, it is not been deeply studied on deep learning service, especially on general-purpose hardware.In this work, we systematically analyze the opportunities and challenges of providing multi-tenant deep learning services on the general-purpose CPU architecture from the aspects of scheduling granularity and code generation. We propose an adaptive granularity scheduling scheme to both guarantee resource usage efficiency and reduce the scheduling conflict rate. We also propose an adaptive compilation strategy, by which we can dynamically and intelligently pick a program with proper exclusive and shared resource usage to reduce overall interference-induced performance loss. Compared to the existing works, our design can serve more requests under the same QoS target in various scenarios (e.g., +71%, +62%, +45% for light, medium, and heavy workloads, respectively), and reduce the averaged query latency by 50%. CCS CONCEPTS• Computer systems organization → Neural networks; Cloud computing; • Computing methodologies → Concurrent algorithms.

show abstract

Section: Multi-tenant Deep Learning Servicementioning

confidence: 99%

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Liu

Leng

Zhang

et al. 2022

Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…I1: Deterministic online execution [28,51]. Different from offline training which could be resource-intensive and last for days or weeks, the inference for each query is often completed with sub-second response time and consumes much less resources.…”

Section: Trainingmentioning

confidence: 99%

“…Reducing the parallelism of execution eliminates the interference from other tasks, but inevitably brings lower throughput and resource utilization. To address this issue, Abacus [29] tries to guarantee SLO for query requests under the GPU co-location scenarios. It controls the execution sequence and the co-location situation proactively, rather than the default random-ordered execution overlap.…”

Section: Efficiencymentioning

confidence: 99%

Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision

Gao¹,

Hu²,

Ye³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning (DL) shows its prosperity in a wide variety of fields. The development of a DL model is a time-consuming and resource-intensive procedure. Hence, dedicated GPU accelerators have been collectively constructed into a GPU datacenter. An efficient scheduler design for such GPU datacenter is crucially important to reduce the operational cost and improve resource utilization. However, traditional approaches designed for big data or high performance computing workloads can not support DL workloads to fully utilize the GPU resources. Recently, substantial schedulers are proposed to tailor for DL workloads in GPU datacenters. This paper surveys existing research efforts for both training and inference workloads. We primarily present how existing schedulers facilitate the respective workloads from the scheduling objectives and resource consumption features. Finally, we prospect several promising future research directions. More detailed summary with the surveyed paper and code links can be found at our project website: https://github.com/S-Lab-System-Group/Awesome-DL-Scheduling-Papers. CCS Concepts: • General and reference → Surveys and overviews; • Computing methodologies → Machine learning; • Computer systems organization → Cloud computing.

show abstract

“…Ebird [59] also proposes a batch-based approach to enable concurrent execution of DNNs with high data transfercompute overlapping. Abacus [60] proposes an operator overlapping strategy based on precise latency prediction. Besides the multi-DNN serving scenario, emerging microservice-based workloads also have complex inner structures similar to DNN models [61], to which our design may also be applied.…”

Section: Multi-tenant Deep Learning Servicementioning

confidence: 99%

VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling

Liu¹,

Leng²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning (DL) models have achieved great success in many application domains. As such, many industrial companies such as Google and Facebook have acknowledged the importance of multi-tenant DL services. Although the multitenant service has been studied in conventional workloads, it is not been deeply studied on deep learning service, especially on general-purpose hardware.In this work, we systematically analyze the opportunities and challenges of providing multi-tenant deep learning services on the general-purpose CPU architecture from the aspects of scheduling granularity and code generation. We propose an adaptive granularity scheduling scheme to both guarantee resource usage efficiency and reduce the scheduling conflict rate. We also propose an adaptive compilation strategy, by which we can dynamically and intelligently pick a program with proper exclusive and shared resource usage to reduce overall interference-induced performance loss. Compared to the existing works, our design can serve more requests under the same QoS target in various scenarios (e.g., +71%, +62%, +45% for light, medium, and heavy workloads, respectively), and reduce the averaged query latency by 50%.

show abstract

Enable simultaneous DNN services based on deterministic operator overlap and precise latency prediction

Cited by 38 publications

References 34 publications

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision

VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling

Contact Info

Product

Resources

About