Jellyfish: Timely Inference Serving for Dynamic Edge Networks

Abstract

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal ? Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

“…ML inference services are user-facing, which mandates high responsiveness [18,37]. Moreover, high accuracy is crucial for these services [20,26]. Consequently, inference systems must deliver highly accurate predictions with fewer computing resources (cost-efficient) while meeting latency constraints under workload variations [18,19,29,37].…”

“…Conversely, overprovisioning wastes computing resources [30,36]. To address these problems caused by dynamic workloads, Autoscaling [2,9,17,18,30,36] resizes the resources of the service, and Model-switching [26,38] switches between ML model variants that differ in their inference latency and accuracy (higher accuracy, higher latency); the former tries to be cost-efficient, and the latter tries to be more accurate, while both guarantee latency SLOs.…”

“…Due to interaction with online users, inference services are latency-sensitive [18,36], and since they contain heavy computations, they are resource-intensive [10]. Accuracy is also a pillar dimension of these services [26]. Faced with dynamic workload [18,37], it is essential to consider the ternary tradeoff space between latency, accuracy, and the resource cost dynamically to address latency requirements while gaining higher accuracy cost-efficiently.…”