Scalable Load Balancing in Networked Systems: Universality Properties and Stochastic Coupling Methods

Abstract

We present an overview of scalable load balancing algorithms which provide favorable delay performance in large-scale systems, and yet only require minimal implementation overhead. Aimed at a broad audience, the paper starts with an introduction to the basic load balancing scenario -referred to as the supermarket model -consisting of a single dispatcher where tasks arrive that must immediately be forwarded to one of N single-server queues. The supermarket model is a dynamic counterpart of the classical balls-and-bins setup where balls must be sequentially distributed across bins.A popular class of load balancing algorithms are so-called power-of-d or JSQ(d) policies, where an incoming task is assigned to a server with the shortest queue among d servers selected uniformly at random. As the name reflects, this class includes the celebrated Join-the-Shortest-Queue (JSQ) policy as a special case (d = N ), which has strong stochastic optimality properties and yields a mean waiting time that vanishes as N grows large for any fixed subcritical load. However, a nominal implementation of the JSQ policy involves a prohibitive communication burden in large-scale deployments. In contrast, a simple random assignment policy (d = 1) does not entail any communication overhead, but the mean waiting time remains constant as N grows large for any fixed positive load.In order to examine the fundamental trade-off between delay performance and implementation overhead, we consider an asymptotic regime where the diversity parameter d(N ) depends on N . We investigate what growth rate of d(N ) is required to match the optimal performance of the JSQ policy on fluid and diffusion scale, and achieve a vanishing waiting time in the limit. The results demonstrate that the asymptotics for the JSQ(d(N )) policy are insensitive to the exact growth rate of d(N ), as long as the latter is sufficiently fast, implying that the optimality of the JSQ policy can asymptotically be preserved while dramatically reducing the communication overhead.Stochastic coupling techniques play an instrumental role in establishing the asymptotic optimality and universality properties, and augmentations of the coupling constructions allow these properties to be extended to infinite-server settings and network scenarios. We additionally show how the communication overhead can be reduced yet further by the so-called Join-the-Idle-Queue (JIQ) scheme, leveraging memory at the dispatcher to keep track of idle servers.In the present paper we review scalable load balancing algorithms (LBAs) which achieve excellent delay performance in large-scale systems and yet only involve low implementation overhead. LBAs play a critical role in distributing service requests or tasks (e.g. compute jobs, data base look-ups, file transfers) among servers or distributed resources in parallel-processing systems. The analysis and design of LBAs has attracted strong attention in recent years, mainly spurred by crucial scalability challenges arising in cloud networks and data centers with massive...

A Simple Steady-State Analysis of Load Balancing Algorithms in the Sub-Halfin-Whitt Regime

“…This seminal work has led to several significant developments: (i) [3] proved that the stationary distribution indeed converges to the stationary distribution of the two-dimensional diffusion limit based on Stein's method; and (ii) via stochastic coupling, [13] showed that the diffusion limit of Pod converges to that of JSQ in the Halfin-Whitt regime at the process level (over finite time) when d = Θ( √ N log N); and (iii) when α < 1/6, [10] proved that the waiting probability of a job is asymptotically zero with d = ω 1 1−λ at the steady-state based on Stein's method. Interested readers can find a comprehensive survey of recent results in [16].…”

Analysis of the shortest relay queue policy in a cooperative random access network with collisions

“…Such schemes improve the system's performance by dividing the work effectively among the participating nodes. Under certain conditions (identical servers, exponential service times, and service protocols that do not depend on the service requirement, such as the FCFS), the so-called join the shortest queue (JSQ) policy has several strong optimality properties: The JSQ policy minimises the overall mean delay among the class of load balancing policies that do not have any advance knowledge of the service requirements [19,40,60].…”

“…The employed cooperation strategy among source and relays is queue-based, with as ultimate goal to minimise the total transmission time, i.e., the time that is needed to transmit a packet from the source to the destination. To this end, we consider join the shortest queue policy as it seems to be the most appropriate for such a wireless network [39,40,60,63].…”

“…Substituting π k,l (defined in (33)) into (55)- (60) and equating the powers of ρ leads to the following recursion equations for the coefficients, for n = 0, 1, . .…”

Job assignment in large-scale service systems with affinity relations

“…The JIQ policy only has a constant communication overhead per job, but requires memory at the dispatcher. We refer to Van der Boor et al [21] and Gamarnik et al [6] for further details.…”