Evaluating high performance communication

Abstract

Recently, high speed interconnects capable of remote direct memory access (RDMA) such as InfiniBand and iWARP have gained considerable popularity due to their superb latency and bandwidth. Most existing studies about RDMA focus on its performance aspect. However, as power management has become essential for high-end systems such as enterprise servers and high performance computing nodes which are often equipped with RDMA capable network adapters, it is very important for us to take a fresh look at the benefits of RDMA from the power perspective.In the paper, we provide a detailed empirical study of the benefits of RDMA in terms of power savings compared with traditional communication protocols such as TCP/IP. We used two popular RDMA adapters in our evaluations: Mellanox ConnectX InfiniBand HCAs and Chelsio T3 10GE RNICs. In order to isolate the impact of communication on power consumption, our evaluation focused on using microbenchmarks which perform different communication patterns. We have also studied several important factors that may have an impact on the performance and the power consumption of RDMA adapters such as the use of polling versus blocking, CPU speeds, and extra memory copies.We show that using high speed RDMA adapters can result in significant amount of power consumption during communication. (In one test, the system power has increased by as much as 50 watts, or over 30% of the idle power.) We found that RDMA generally has better power efficiency compared to that of TCP/IP, especially for communication intensive phases, for example when large messages are transferred. The power savings of RDMA are achieved by minimizing the interactions between the network adapters and other system components such as the CPUs and the memory: Although nearly the same amount of data must be going through the network adapters for both RDMA and TCP/IP, RDMA rePermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. quires much fewer CPU cycles for protocol processing and also generates less memory bus traffic, both of which contribute to its power savings.Overall, our research demonstrated that RDMA not only provides high communication performance, but also offers excellent power efficiency, making it a desirable choice in environments that have strict power/energy constraints and demand high communication performance.

Per-call Energy Saving Strategies in All-to-All Communications

“…3(left)) with the Proposed strategy. Note, however that, in all the cases, the power consumption increases with the message size since the memory dynamic power consumption increases because of message copying [13]. Similar power reductions have been obtained for the all-to-all-vector operation.…”

“…However, introducing just one idle cycle per eight cycles degrades the intranode communication considerably (about 25%). This is expected since intranode communication uses more CPU cycles for a message transfer whereas in internode transfers RDMA offloads a large part of the communication processing to the NICs [13].…”

“…In [13], authors study the power efficiency and communication performance of RDMA over TCP/IP and conclude that RDMA performs better for the both cases. The energy efficiency delivered by the modern interconnects in high performance clusters is discussed in [4].…”

Runtime Procedure for Energy Savings in Applications with Point-to-Point Communications

“…Then a (user-defined) performance loss γ must be considered to constrain the amount of the performance degradation that is tolerated when the CPU frequency is scaled down. In [21], an appropriate frequency is selected for a single communication call by taking into account just the message size while exploiting the Infiniband CPU offload feature [13] and the communication characteristics. In the case of a multicall phase, the gaps between call pairs must be examined to evaluate their performance losses on a lower frequency.…”

Designing Energy Efficient Communication Runtime Systems for Data Centric Programming Models

“…Researchers in [42] and [43] propose energy saving approaches using DVFS and CPU throttling for collective communication primitives. Liu et al, have provided a detailed empirical study of the benefits of power efficiency of RDMA compared to the traditional communication protocols such as TCP/IP [44]. However, this work has been done using verbs level interface, and does not provide guidance for higher level communication protocols for implementation.…”