ENTS: An Edge-native Task Scheduling System for Collaborative Edge Computing

Abstract

Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively. Experiments of Llama2 serial models on a heterogeneous physical prototype demonstrate that EdgeShard achieves up to 50% latency reduction and 2x throughput improvement over baseline methods.

Task Allocation Methods and Optimization Techniques in Edge Computing: A Systematic Review of the Literature

“…Context-aware task allocation [52] DFD DAG [69] GT IoT, 5G [78] LPA MEC Energy-efficient task allocation [68] MAPE-K IoT [11] Lyapunov MEC [19] Bi-Level MEC [25] DRL IoV [30] QT 5G [43] PSO IoT [47] ILP MEC [71] JTORA MEC [73] hybrid RF-FSO Industrial IoT Dynamic task allocation [13] MPSO VVECNs, VANETs [79] MH IoV [20] AA Fog [23] DP MEC [41] DRL IoT [44] ECTA EC [46] AA Fog [49] ILP [56] GA, GEN IoT [60] PSO EC [64] BPSO Table 1. Cont.…”

“…A recent paper by Mingjin Zhang et al [49] proposed an entirely novel edge-native task scheduling system (ENTS) that co-schedules networking and computing resources to improve the performance of edge-native applications. Although Kubernetes is now the de facto standard for container orchestration, it does not enable edge-native apps or take into consideration their unique throughput and latency needs.…”

A survey of Kubernetes scheduling algorithms

“…As the computing paradigm moves towards edge and fog computing, Kubernetes is proving to be a versatile solution that provides seamless network management between cloud and edge nodes [2][3][4]. Kubernetes face multiple challenges when deployed in an IoT environment.…”

“…Kubernetes face multiple challenges when deployed in an IoT environment. These challenges range from optimizing network traffic distribution [2], optimizing flow routing policies [3], and edge device's computational resources distribution [4].…”