Swayam

Abstract

Developers use Machine Learning (ML) platforms to train ML models and then deploy these ML models as web services for inference (prediction). A key challenge for platform providers is to guarantee response-time Service Level Agreements (SLAs) for inference workloads while maximizing resource e ciency. Swayam is a fully distributed autoscaling framework that exploits characteristics of production ML inference workloads to deliver on the dual challenge of resource e ciency and SLA compliance. Our key contributions are (1) model-based autoscaling that takes into account SLAs and ML inference workload characteristics, (2) a distributed protocol that uses partial load information and prediction at frontends to provision new service instances, and (3) a backend self-decommissioning protocol for service instances. We evaluate Swayam on 15 popular services that were hosted on a production ML-as-a-service platform, for the following service-speci c SLAs: for each service, at least 99% of requests must complete within the response-time threshold. Compared to a clairvoyant autoscaler that always satis es the SLAs (i.e., even if there is a burst in the request rates), Swayam decreases resource utilization by up to 27%, while meeting the service-speci c SLAs over 96% of the time during a three hour window. Microsoft Azure's Swayam-based framework was deployed in 2016 and has hosted over 100,000 services.

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

“…MS [38] INFaaS [30] Cocktail [20] VPA [9] InfAdapter Cost Optimization ✕ ✓ ✓ * ✓ ✓ Accuracy Maximization ✓ ✕ ✓ ✕ ✓ Predictive Decision-Making ✓ ✕ ✓ ✓ ✓ Container as a Service (CaaS) ✕ ✕ ✕ ✓ ✓ Latency SLO-aware ✓ ✓ ✓ ✕ ✓ machine translation, chatbots, medical, and recommender systems, are running in data centers [13,28,32,34], comprising more than 90% of computing resources allocated to ML [10,13,25]. ML inference services are user-facing, which mandates high responsiveness [18,37]. Moreover, high accuracy is crucial for these services [20,26].…”

“…Moreover, high accuracy is crucial for these services [20,26]. Consequently, inference systems must deliver highly accurate predictions with fewer computing resources (cost-efficient) while meeting latency constraints under workload variations [18,19,29,37].…”

“…The dynamic nature of inference serving workloads requires different resource allocations for ML services [18,37]. Failing to right-size the services results in over or underresource provisioning.…”

“…Failing to right-size the services results in over or underresource provisioning. Under-provisioning leads to service level objective (SLO) violations (e.g., 99 𝑡ℎ percentile of latency distribution, P99-latency) [18,36]. Conversely, overprovisioning wastes computing resources [30,36].…”

“…Conversely, overprovisioning wastes computing resources [30,36]. To address these problems caused by dynamic workloads, Autoscaling [2,9,17,18,30,36] resizes the resources of the service, and Model-switching [26,38] switches between ML model variants that differ in their inference latency and accuracy (higher accuracy, higher latency); the former tries to be cost-efficient, and the latter tries to be more accurate, while both guarantee latency SLOs.…”