RDMA over Commodity Ethernet at Scale

Guo, Chuanxiong; Wu, Haitao; Zhong, Dan; Soni, Gaurav; Ye, Jianxi; Padhye, Jitendra; Lipshteyn, Marina

doi:10.1145/2934872.2934908

Cited by 268 publications

(112 citation statements)

References 25 publications

Supporting

Mentioning

110

Contrasting

Order By: Relevance

“…Any degradation in performance due to disabling PFC with IRN stayed within 9%. Incast with cross traffic: In practice we expect incast to occur with other cross traffic in the network [23,29]. We started an incast as described above with M = 30, along with our default case workload running at 50% link utilization level.…”

Section: Incastmentioning

confidence: 99%

Revisiting network support for RDMA

Mittal

Shpiner

Panda³

et al. 2018

Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

120

View full text Add to dashboard Cite

The advent of RoCE (RDMA over Converged Ethernet) has led to a significant increase in the use of RDMA in datacenter networks. To achieve good performance, RoCE requires a lossless network which is in turn achieved by enabling Priority Flow Control (PFC) within the network. However, PFC brings with it a host of problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks. Rather than seek to fix these issues, we instead ask: is PFC fundamentally required to support RDMA over Ethernet?We show that the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement. We propose an improved RoCE NIC (IRN) design that makes a few simple changes to the RoCE NIC for better handling of packet losses. We show that IRN (without PFC) outperforms RoCE (with PFC) by 6-83% for typical network scenarios. Thus not only does IRN eliminate the need for PFC, it improves performance in the process! We further show that the changes that IRN introduces can be implemented with modest overheads of about 3-10% to NIC resources. Based on our results, we argue that research and industry should rethink the current trajectory of network support for RDMA.

show abstract

Section: Incastmentioning

confidence: 99%

Revisiting network support for RDMA

Mittal

Shpiner

Panda³

et al. 2018

Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

120

View full text Add to dashboard Cite

show abstract

“…The technical differences between TCP and RDMA are further explained in Appendix . TCP is known to be inefficient for HPC applications, and various non‐TCP mechanisms have been developed to improve the communication performance even on Ethernet clusters (Gavrilovska, 2009), such as RDMA over Ethernet (known as RoCE, Guo et al, 2016). Although AWS EC2 does not yet support RDMA, the newly introduced AWS EFA provides a high‐performance, non‐TCP communication mechanism, supporting some RDMA‐like features (Appendix ).…”

Section: Benchmarking Internode Communication Performancementioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

“…Confining the fabric to a bounded-size unit avoids the emergent safety, performance and monitoring challenges of large-scale fabrics. For example, recent work has shown that scaling RDMA over commodity Ethernet introduces issues of congestion control, dealing with deadlocks and livelocks, and other subtleties of priority-based flow control [29,61].…”

Section: Rackout Data Servingmentioning

confidence: 99%

The Case for RackOut

Novaković¹,

Daglis²,

Bugnion³

et al. 2016

Proceedings of the Seventh ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

To provide low latency and high throughput guarantees, most large key-value stores keep the data in the memory of many servers. Despite the natural parallelism across lookups, the load imbalance, introduced by heavy skew in the popularity distribution of keys, limits performance. To avoid violating tail latency service-level objectives, systems tend to keep server utilization low and organize the data in micro-shards, which provides units of migration and replication for the purpose of load balancing. These techniques reduce the skew, but incur additional monitoring, data replication and consistency maintenance overheads.In this work, we introduce RackOut, a memory pooling technique that leverages the one-sided remote read primitive of emerging rack-scale systems to mitigate load imbalance while respecting service-level objectives. In RackOut, the data is aggregated at rack-scale granularity, with all of the participating servers in the rack jointly servicing all of the rack's micro-shards. We develop a queuing model to evaluate the impact of RackOut at the datacenter scale. In addition, we implement a RackOut proof-of-concept key-value store, evaluate it on two experimental platforms based on RDMA and Scale-Out NUMA, and use these results to validate the model. Our results show that RackOut can increase throughput up to 6× for RDMA and 8.6× for Scale-Out NUMA compared to a scale-out deployment, while respecting tight tail latency service-level objectives.

show abstract

RDMA over Commodity Ethernet at Scale

Cited by 268 publications

References 25 publications

Revisiting network support for RDMA

Revisiting network support for RDMA

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

The Case for RackOut

Contact Info

Product

Resources

About