Scalable load balancing in networked systems: A survey of recent advances

Abstract

The basic load balancing scenario involves a single dispatcher where tasks arrive that must immediately be forwarded to one of N single-server queues. We discuss recent advances on scalable load balancing schemes which provide favorable delay performance when N grows large, and yet only require minimal implementation overhead.Join-the-Shortest-Queue (JSQ) yields vanishing delays as N grows large, as in a centralized queueing arrangement, but involves a prohibitive communication burden. In contrast, power-of-d or JSQ(d) schemes that assign an incoming task to a server with the shortest queue among d servers selected uniformly at random require little communication, but lead to constant delays. In order to examine this fundamental trade-off between delay performance and implementation overhead, we consider JSQ(d(N)) schemes where the diversity parameter d(N) depends on N and investigate what growth rate of d(N) is required to asymptotically match the optimal JSQ performance on fluid and diffusion scale.Stochastic coupling techniques and stochastic-process limits play an instrumental role in establishing the asymptotic optimality. We demonstrate how this methodology carries over to infinite-server settings, finite buffers, multiple dispatchers, servers arranged on graph topologies, and token-based load balancing including the popular Join-the-Idle-Queue (JIQ) scheme. In this way we provide a broad overview of the many recent advances in the field. This survey extends the short review presented at ICM 2018 [121].

Utility maximizing load balancing policies

“…In large-scale deployments, this overhead has two main sources: the communication burden of messaging between the dispatcher and the servers, and the operational cost of storing and managing state information at the dispatcher [8,9]. We refer to [3] for an extensive survey on scalable load balancing.…”

“…Figure 1: Schematic view of some of the related work. Most of the load balancing literature concerns systems of parallel and homogeneous single-server queues; this vast literature is surveyed in [3]. Some recent papers study single-server dynamics in heterogeneous settings or infinite-server dynamics in homogeneous settings, whereas the present paper considers a heterogeneous system with infinite-server dynamics.…”

“…A server pool has coordinates (i, j) ∈ I + if its class is i and it has precisely j − 1 tasks; e.g., in Figure 2, server pool A of class 1 has coordinates (1,4) and both server pools of class 3 have coordinates (3,1). Since server pools with the same coordinates are statistically identical, we may focus on ranking coordinates rather than server pools.…”

Stability and Optimization of Speculative Queueing Networks

“…Over the last decade there has been interest in the performance analysis of policies that dispatch to the shortest among a set of queues, for instance, power-of-two-choices and join-the-idle-queue; see [40] for a survey. However, more recently there has been an increased focus on job replication and in the following, we review existing replication approaches; for further details, we refer the reader to the recent survey [25].…”

Load Balancing in Heterogeneous Server Clusters: Insights From a Product-Form Queueing Model

“…Besides static approaches that do not require communication between the dispatcher and the servers, classical solutions include join-the-shortest-queue, power-of-d-choices [19], and join-idle-queue [18]. These solutions were originally designed for service systems that are homogeneous in the sense that all servers have the same speed [5]. Although these solutions successfully cope with the stochastic nature of demand, they are not always suitable when servers have unequal speeds [10].…”