Heterogeneity and Interference-Aware Virtual Machine Provisioning for Predictable Performance in the Cloud

Abstract

GPUs are essential to accelerating the latency-sensitive deep neural network (DNN) inference workloads in cloud datacenters. To fully utilize GPU resources, spatial sharing of GPUs among co-located DNN inference workloads becomes increasingly compelling. However, GPU sharing inevitably brings severe performance interference among co-located inference workloads, as motivated by an empirical measurement study of DNN inference on EC2 GPU instances. While existing works on guaranteeing inference performance service level objectives (SLOs) focus on either temporal sharing of GPUs or reactive GPU resource scaling and inference migration techniques, how to proactively mitigate such severe performance interference has received comparatively little attention. In this paper, we propose iGniter, an interference-aware GPU resource provisioning framework for cost-efficiently achieving predictable DNN inference in the cloud. iGniter is comprised of two key components: (1) a lightweight DNN inference performance model, which leverages the system and workload metrics that are practically accessible to capture the performance interference; (2) A cost-efficient GPU resource provisioning strategy that jointly optimizes the GPU resource allocation and adaptive batching based on our inference performance model, with the aim of achieving predictable performance of DNN inference workloads. We implement a prototype of iGniter based on the NVIDIA Triton inference server hosted on EC2 GPU instances. Extensive prototype experiments on four representative DNN models and datasets demonstrate that iGniter can guarantee the performance SLOs of DNN inference workloads with practically acceptable runtime overhead, while saving the monetary cost by up to 25% in comparison to the state-of-the-art GPU resource provisioning strategies.

Joint Information Freshness and Completion Time Optimization for Vehicular Networks

“…Constraint (17) gives the load intensity at the second queue. Constraints (18) and (19) give the expression for the expected completion time and expected AoI for all jobs j, respectively. Finally, Constraint (22) ensures the positivity of the scheduling probabilities.…”

Managing energy, performance and cost in large scale heterogeneous datacenters using migrations

“…Applications that are running inside VMs are affected by many factors including virtualisation and co-allocation [17]. In the literature [20], [21], [22], [23], different approaches have been suggested to manage datacenter resources in an energy, performance and cost-efficient way. These can be categorized as: (i) resource provisioning; (ii) consolidation with migration; and (iii) methods that describe the trade-off between energy, performance and cost.…”

A Lion‐Whale optimization‐based migration of virtual machines for data centers in cloud computing

“…Xu et al have introduced a heterogeneity and interference‐aware VM provisioning framework (Heifer) for tenant applications by focusing on MapReduce as a representative cloud application. This framework calculated the MapReduce application's performance by developing a lightweight performance model based on the online‐measured resource utilization and capturing VM interference.…”