Flow control for Latency-Critical RPCs

Abstract

In today's modern datacenters, the waiting time spent within a server's queue is a major contributor of the end-to-end tail latency of µs-scale remote procedure calls. In traditional TCP, congestion control handles in-network congestion, while flow control was designed to avoid memory overruns in streaming scenarios. The latter is unfortunately oblivious to the load on the server when processing short requests from multiple clients at very high rates. Acknowledging flow control as the mechanism that controls queuing on the end-host, we propose a different flow control mechanism that depends on the application-specific service-level objectives and controls the waiting time in the receivers queue by adjusting the incoming load accordingly. We design this latency-aware flow control mechanism as part of TCP by maintaining a wire-compatible header format without introducing extra messages. We implement a proof-of-concept userspace TCP stack on top of DPDK and we show that the new flow control mechanism prevents applications from violating service-level objectives in a single-server environment by throttling the incoming requests. We demonstrate the true benefit of the approach in a replicated, multi-server scenario, where independent clients leverage the flow-control signal to avoid directing requests to the overloaded servers.

A Graph Attention Mechanism-Based Multiagent Reinforcement-Learning Method for Task Scheduling in Edge Computing

“…The load generated by devices tends to follow certain rules (e.g., Poisson inter-arrival time distribution [37]). At the beginning of each time slice, the end device possesses multiple tasks k m (t) and the data size of k m (t) obeys the uniform distribution.…”

The NEBULA RPC-Optimized Architecture

“…Provisioning a receive buffer of limited size on the server requires the transport protocol to signal a "failure to deliver" (NACK) if the request is dropped because of a full queue. It is up to the client to react to a NACK reception; for example, the request could be retried or sent to a different server, as proposed by Kogias et al [45]. Exposing delivery failures to the client follows the end-to-end principle in systems design [46]: the client application is best equipped to handle such violations and should be informed immediately.…”

“…The need for low latency has led systems designers to aggressively limit queuing in the transport and RPC protocol stacks themselves [45], [59]. Kogias et al [45] also observe that limiting server-side queuing is critical for µs-scale RPCs, and use TCP flow control to limit the number of requests per connection based on the application's SLO. NEBULA performs buffer management for hardware-rather than software-terminated protocols.…”