Unlocking Credit Loop Deadlocks

Shpiner, Alex; Zahavi, Eitan; Zdornov, Vladimir; Anker, Tal; Kadosh, Matty

doi:10.1145/3005745.3005768

Cited by 21 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is primarily because HPC clusters are smaller with more controlled traffic patterns, and hence the negative effects of providing losslessness (such as congestion spreading and deadlocks) are rarer. PFC's issues are exacerbated on larger scale clusters [23,24,29,34,37]. Credit-based Flow Control: Since the focus of our work was RDMA deployment over Ethernet, our experiments used PFC.…”

Section: Discussion and Related Workmentioning

confidence: 99%

“…However, the current solution is not without problems. In particular, PFC adds management complexity and can lead to significant performance problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks [23,24,34,36,37]. Rather than continue down the current path and address the various problems with PFC, in this paper we take a step back and ask whether it was needed in the first place.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Revisiting network support for RDMA

Mittal

Shpiner

Panda³

et al. 2018

Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Self Cite

138

View full text Add to dashboard Cite

The advent of RoCE (RDMA over Converged Ethernet) has led to a significant increase in the use of RDMA in datacenter networks. To achieve good performance, RoCE requires a lossless network which is in turn achieved by enabling Priority Flow Control (PFC) within the network. However, PFC brings with it a host of problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks. Rather than seek to fix these issues, we instead ask: is PFC fundamentally required to support RDMA over Ethernet?We show that the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement. We propose an improved RoCE NIC (IRN) design that makes a few simple changes to the RoCE NIC for better handling of packet losses. We show that IRN (without PFC) outperforms RoCE (with PFC) by 6-83% for typical network scenarios. Thus not only does IRN eliminate the need for PFC, it improves performance in the process! We further show that the changes that IRN introduces can be implemented with modest overheads of about 3-10% to NIC resources. Based on our results, we argue that research and industry should rethink the current trajectory of network support for RDMA.

show abstract

Section: Discussion and Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Revisiting network support for RDMA

Mittal

Shpiner

Panda³

et al. 2018

Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Self Cite

138

View full text Add to dashboard Cite

show abstract

“…As expected, Figure 6 shows only a constant amount of data traverses the network when using RMCs to compute hashes at the server. Prior work [27,35,48] has discussed many of the challenges of using RDMA in congested networks. In this case, the use of RMCs reduces the amount of bulk data transferred over the network fabric, thereby reducing congestion.…”

Section: Hashing a Remote Buffermentioning

confidence: 99%

Remote Memory Calls

Amaro

Luo

Ousterhout

et al. 2020

Proceedings of the 19th ACM Workshop on Hot Topics in Networks

View full text Add to dashboard Cite

show abstract

“…PFC pauses all upstream interfaces once it detects a risk of packet loss, and the pauses can propagate via a tree-like graph to multiple hops away. Such spreading of congestion can possibly trigger PFC deadlocks [21,23,38] and PFC storms (Case-1 in §1) that can silence a lot of senders even if the network has free capacity. Despite the probability of PFC deadlocks and storms being fairly small, they are still big threats to operators and applications, since currently we have no methods to guarantee they won't occur [23].…”

Section: Our Goals For Rdmamentioning

confidence: 99%

HPCC

Miao

Liu

et al. 2019

Proceedings of the ACM Special Interest Group on Data Communication

331

View full text Add to dashboard Cite

Congestion control (CC) is the key to achieving ultra-low latency, high bandwidth and network stability in high-speed networks. From years of experience operating large-scale and high-speed RDMA networks, we find the existing high-speed CC schemes have inherent limitations for reaching these goals. In this paper, we present HPCC (High Precision Congestion Control), a new high-speed CC mechanism which achieves the three goals simultaneously. HPCC leverages in-network telemetry (INT) to obtain precise link load information and controls traffic precisely. By addressing challenges such as delayed INT information during congestion and overreaction to INT information, HPCC can quickly converge to utilize free bandwidth while avoiding congestion, and can maintain near-zero in-network queues for ultra-low latency. HPCC is also fair and easy to deploy in hardware. We implement HPCC with commodity programmable NICs and switches. In our evaluation, compared to DCQCN and TIMELY, HPCC shortens flow completion times by up to 95%, causing little congestion even under large-scale incasts. CCS CONCEPTS• Networks → Transport protocols; Data center networks;

show abstract

Unlocking Credit Loop Deadlocks

Cited by 21 publications

References 10 publications

Revisiting network support for RDMA

Revisiting network support for RDMA

Remote Memory Calls

HPCC

Contact Info

Product

Resources

About