TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep Learning Inference in Function-as-a-Service

Abstract

Deep neural networks (DNNs) have become core computation components within low latency Function as a Service (FaaS) prediction pipelines: including image recognition, object detection, natural language processing, speech synthesis, and personalized recommendation pipelines. Cloud computing, as the de-facto backbone of modern computing infrastructure for both enterprise and consumer applications, has to be able to handle user-defined pipelines of diverse DNN inference workloads while maintaining isolation and latency guarantees, and minimizing resource waste. The current solution for guaranteeing isolation within FaaS is suboptimal -suffering from "cold start" latency. A major cause of such inefficiency is the need to move large amount of model data within and across servers. We propose TrIMS as a novel solution to address these issues. Our proposed solution consists of a persistent model store across the GPU, CPU, local storage, and cloud storage hierarchy, an efficient resource management layer that provides isolation, and a succinct set of application APIs and container technologies for easy and transparent integration with FaaS, Deep Learning (DL) frameworks, and user code. We demonstrate our solution by interfacing TrIMS with the Apache MXNet framework and demonstrate up to 24× speedup in latency for image classification models and up to 210× speedup for large models. We achieve up to 8× system throughput improvement.

Trustworthy artificial intelligence

“…Electronic markets, for example, are increasingly augmented with AI-based systems such as customer service chatbots (Adam et al 2020). Likewise, several cloud providers recently began offering 'AI as a Service' (AIaaS), referring to web services for organizations and individuals interested in training, building, and deploying AI-based systems (Dakkak et al 2019;Rai et al 2019). Although cost-and time-saving opportunities have triggered a widespread implementation of AI-based systems and services in electronic markets, trust persists to play a pivotal role in any buyer-seller relationship (Bauer et al 2019;Marella et al 2020).…”

Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision

“…The RL model takes as input the Experiment GPU Scales: the scale of physical testbed. S (0, 30] M (30,60] L (60, 120] XL (120, ∞] -: no evaluation on a physical cluster or not clearly specified. job time information, resource demand, and accuracy requirements.…”

Impact of Artificial Intelligence-enabled Software-defined Networks in Infrastructure and Operations: Trends and Challenges

“…In contrast, with cloud services like IaaS and PaaS, the abstraction is done for hardware and operating system, respectively. Whereas in serverless computing, the abstraction is done at language runtime, and everything is provided using functions, so it gives function as a Service (FaaS) [19]. Fig 2 shows the user's control over the cloud services.…”