Congestion Control for Large-Scale RDMA Deployments

Zhu, Yibo; Eran, Haggai; Firestone, Daniel; Guo, Chuanxiong; Lipshteyn, Marina; Liron, Yehonatan; Padhye, Jitendra; Raindel, Shachar; Yahia, Mohamad Haj; Zhang, Ming

doi:10.1145/2785956.2787484

Cited by 258 publications

(72 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing window-based congestion control schemes are not effective for this particular type of workload consisting of very small messages [6] that fit in a single packet, across a large number of connections, in an environment with µs RTTs. Congestion control for such workloads is an open research problem [11,12,31,47] with recent research proposals [33] suggesting a generic congestion control agent that can be used in kernel-bypass networking stacks. Such an approach is a good fit for our networking stack, but we leave this for future work.…”

Section: Methodsmentioning

confidence: 99%

Flow control for Latency-Critical RPCs

Kogias

Bugnion

2018

Proceedings of the 2018 Afternoon Workshop on Kernel Bypassing Networks

View full text Add to dashboard Cite

In today's modern datacenters, the waiting time spent within a server's queue is a major contributor of the end-to-end tail latency of µs-scale remote procedure calls. In traditional TCP, congestion control handles in-network congestion, while flow control was designed to avoid memory overruns in streaming scenarios. The latter is unfortunately oblivious to the load on the server when processing short requests from multiple clients at very high rates. Acknowledging flow control as the mechanism that controls queuing on the end-host, we propose a different flow control mechanism that depends on the application-specific service-level objectives and controls the waiting time in the receivers queue by adjusting the incoming load accordingly. We design this latency-aware flow control mechanism as part of TCP by maintaining a wire-compatible header format without introducing extra messages. We implement a proof-of-concept userspace TCP stack on top of DPDK and we show that the new flow control mechanism prevents applications from violating service-level objectives in a single-server environment by throttling the incoming requests. We demonstrate the true benefit of the approach in a replicated, multi-server scenario, where independent clients leverage the flow-control signal to avoid directing requests to the overloaded servers.

show abstract

Section: Methodsmentioning

confidence: 99%

Flow control for Latency-Critical RPCs

Kogias

Bugnion

2018

Proceedings of the 2018 Afternoon Workshop on Kernel Bypassing Networks

View full text Add to dashboard Cite

show abstract

“…InfiniBand provides rate-based end-to-end congestion control using ECN marks [10,22]. DCQCN [55] has shown that RoCE without end -to-end congestion control degrades in both latency and throughput at high loads.…”

Section: Challengesmentioning

confidence: 99%

“…Unlike TCP's window-based rate control, RCP's [16] routers iteratively calculate and directly convey the fair-share bandwidth to the senders sharing a link. DCQCN [55] and TIMELY [38] improve end-to-end congestion control at datacenter scales for RDMA (RoCE) and user-level TCP respectively. Both DCQCN and TIMELY directly control the sending rate by pacing the packets sent out of the NIC.…”

Section: Challengesmentioning

confidence: 99%

“…InfiniBand uses Early Congestion Notification (ECN) marks to infer imminent congestion and cuts back the sending rates [10,22]. While DCQCN [55] proposes a similar scheme for RoCE, TIMELY [38] uses round-trip times (RTT) measurements, instead of ECN marks, for rate control in user-level TCP. Unfortunately, because ECN marks and RTT measurements need many round-trips to converge to the appropriate sending rates (e.g., 50 RTTs in TIMELY), the schemes are too slow for the applications' predominantly short flows each of which lasts only a handful of round-trips.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dart: Divide and Specialize for Fast Response to Congestion in RDMA-Based Datacenter Networks

Xue

Chaudhry

Vamanan

et al. 2020

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

Though Remote Direct Memory Access (RDMA) promises to reduce datacenter network latencies significantly compared to TCP (e.g., 10x), end-to-end congestion control in the presence of incasts is a challenge. Targeting the full generality of the congestion problem, previous schemes rely on slow, iterative convergence to the appropriate sending rates (e.g., TIMELY takes 50 RTTs). Several papers have shown that even in oversubscribed datacenter networks most congestion occurs at the receiver. Accordingly, we propose a divide-and-specialize approach, called Dart, which isolates the common case of receiver congestion and further sub-divides the remaining in-network congestion into the simpler spatially-localized and the harder spatially-dispersed cases. For receiver congestion, we propose direct apportioning of sending rates (DASR) in which a receiver for n senders directs each sender to cut its rate by a factor of n, converging in only one RTT. For the spatially-localized case, Dart provides fast (under one RTT) response by adding novel switch hardware for in-order flow deflection (IOFD) because RDMA disallows packet reordering on which previous load balancing schemes rely. For the uncommon spatially-dispersed case, Dart falls back to DCQCN. Small-scale testbed measurements and at-scale simulations, respectively, show that Dart achieves 60% (2.5x) and 79% (4.8x) lower 99 th -percentile latency, and similar and 58% higher throughput than InfiniBand, and TIMELY and DCQCN.

show abstract

“…Confining the fabric to a bounded-size unit avoids the emergent safety, performance and monitoring challenges of large-scale fabrics. For example, recent work has shown that scaling RDMA over commodity Ethernet introduces issues of congestion control, dealing with deadlocks and livelocks, and other subtleties of priority-based flow control [29,61].…”

Section: Rackout Data Servingmentioning

confidence: 99%

The Case for RackOut

Novaković¹,

Daglis²,

Bugnion³

et al. 2016

Proceedings of the Seventh ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

To provide low latency and high throughput guarantees, most large key-value stores keep the data in the memory of many servers. Despite the natural parallelism across lookups, the load imbalance, introduced by heavy skew in the popularity distribution of keys, limits performance. To avoid violating tail latency service-level objectives, systems tend to keep server utilization low and organize the data in micro-shards, which provides units of migration and replication for the purpose of load balancing. These techniques reduce the skew, but incur additional monitoring, data replication and consistency maintenance overheads.In this work, we introduce RackOut, a memory pooling technique that leverages the one-sided remote read primitive of emerging rack-scale systems to mitigate load imbalance while respecting service-level objectives. In RackOut, the data is aggregated at rack-scale granularity, with all of the participating servers in the rack jointly servicing all of the rack's micro-shards. We develop a queuing model to evaluate the impact of RackOut at the datacenter scale. In addition, we implement a RackOut proof-of-concept key-value store, evaluate it on two experimental platforms based on RDMA and Scale-Out NUMA, and use these results to validate the model. Our results show that RackOut can increase throughput up to 6× for RDMA and 8.6× for Scale-Out NUMA compared to a scale-out deployment, while respecting tight tail latency service-level objectives.

show abstract

Congestion Control for Large-Scale RDMA Deployments

Cited by 258 publications

References 26 publications

Flow control for Latency-Critical RPCs

Flow control for Latency-Critical RPCs

Dart: Divide and Specialize for Fast Response to Congestion in RDMA-Based Datacenter Networks

The Case for RackOut

Contact Info

Product

Resources

About