High performance RDMA-based MPI implementation over InfiniBand

Liu, Jiuxing; Wu, Jiesheng; Kini, Sushmitha P.; Wyckoff, P.; Panda, Dhabaleswar K.

doi:10.1145/782852.782855

Cited by 36 publications

(46 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recent work from the systems and networking communities, for example, has focused on RDMA reads as a building block, because they bypass the remote network stack and CPU entirely for GETs [21,8]. In contrast, however, the HPC community has made wider use of messaging, both for key-value caches [14] and general communication [16]. These latter systems scaled to thousands of machines, but provided low throughput-less than one million operations per second in memcached [14].…”

Section: Design Decisionsmentioning

confidence: 99%

“…A PUT request contains a 16-byte keyhash, a 2-byte LEN field (specifying the value's length), and up to 1000 bytes for the value. To poll for incoming requests, we use the left- Figure 8: Layout of the request region at the server to-right ordering of the RNIC's DMA writes [16,8]. We use the keyhash field to poll for new requests; therefore, the key is written to the rightmost 16 bytes of the 1 KB slot.…”

Section: Requestsmentioning

confidence: 99%

“…They favor WRITEs over READs for the same reasons as in our work, suggesting that the performance gap has existed over several generations of InfiniBand hardware. There have been several versions of MPI over InfiniBand [16,19]. MPICH2 uses RDMA writes for one-sided messaging: the server polls the head of a circular buffer that is written to by a client.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Using RDMA efficiently for key-value services

Kalia

Kaminsky

Andersen

2014

SIGCOMM Comput. Commun. Rev.

134

143

View full text Add to dashboard Cite

This paper describes the design and implementation of HERD, a keyvalue system designed to make the best use of an RDMA network. Unlike prior RDMA-based key-value systems, HERD focuses its design on reducing network round trips while using efficient RDMA primitives; the result is substantially lower latency, and throughput that saturates modern, commodity RDMA hardware. HERD has two unconventional decisions: First, it does not use RDMA reads, despite the allure of operations that bypass the remote CPU entirely. Second, it uses a mix of RDMA and messaging verbs, despite the conventional wisdom that the messaging primitives are slow. A HERD client writes its request into the server's memory; the server computes the reply. This design uses a single round trip for all requests and supports up to 26 million key-value operations per second with 5 µs average latency. Notably, for small key-value items, our full system throughput is similar to native RDMA read throughput and is over 2X higher than recent RDMA-based keyvalue systems. We believe that HERD further serves as an effective template for the construction of RDMA-based datacenter services.

show abstract

Section: Design Decisionsmentioning

confidence: 99%

Section: Requestsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Using RDMA efficiently for key-value services

Kalia

Kaminsky

Andersen

2014

SIGCOMM Comput. Commun. Rev.

134

143

View full text Add to dashboard Cite

show abstract

“…RDMA can be implemented over different link layer protocols. It was originally associated with InfiniBand [35], requiring special-purpose interconnect. Two recent Ethernet-base alternatives-RoCE and iWARP-provide a more cost-efficient solution that is expected to increase RDMA adoption in the data center domain.…”

Section: Nvme-over-fabricsmentioning

confidence: 99%

Performance Characterization of NVMe-over-Fabrics Storage Disaggregation

Guz

Shayesteh

et al. 2018

ACM Trans. Storage

View full text Add to dashboard Cite

Storage disaggregation separates compute and storage to different nodes to allow for independent resource scaling and, thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-based SSD) disaggregation is considered more challenging. This is because SSDs are significantly faster than hard drives, so the latency overheads (due to both network and CPU processing) as well as the extra compute cycles needed for the offloading stack become much more pronounced.In this work, we characterize the overheads of NVMe-SSD disaggregation. We show that NVMe-over-Fabrics (NVMe-oF)-a recently released remote storage protocol specification-reduces the overheads of remote access to a bare minimum, thus greatly increasing the cost-efficiency of Flash disaggregation. Specifically, while recent work showed that SSD storage disaggregation via iSCSI degrades application-level throughput by 20%, we report on negligible performance degradation with NVMe-oF-both when using stress-tests as well as with a more-realistic KV-store workload. 31:2 Z. Guz et al. Fig. 1. NVMe-over-Fabrics Architecture.40 times better than SATA-SSDs. Indeed, many large-scale cloud companies have reported using PCIe-based SSDs as part of their infrastructure [19,31,49,57].Generally, storage devices can either be co-located within the compute server nodes, or be placed in dedicated storage nodes accessed through the network. Large-scale cloud companies originally used scale-out infrastructure, composing their data centers out of commodity servers that tightly coupled memory, storage, and compute [3,38]. Unfortunately, this approach leads to inefficiencies and resource underutilization, because it fixes the ratio between compute, memory, storage, and network.Resource underutilization in data centers is a common, well-documented phenomenon [3,31,36,49]. Since the usage of different resources changes over time, predominantly independently from one another, there is no single static resource balance that fits every application that a server supports throughout its lifetime. Since changing these ratios at scale is economically unfeasible [51], server resources are often over-provisioned, leading to an increased total cost of ownership [3]. Specifically, NVMe-SSDs tend to be over-provisioned in terms of both load (IOPS) and capacity: capacity is over-provisioned to allow for future growth, and load is underutilized, because other software overheads tend to saturate the CPU well before reaching the drive limits [11,49]. Indeed, PCIe-based Flash has been argued to be "problematically fast" [64].Resource disaggregation is a common approach to mitigate the problem of over-provisioning. Specifically, storage disaggregation decouples compute and storage to different nodes (i.e., different servers), allowing independent scaling of each resource according to dynamic needs. It provides more flexibility when tuning the infrastructure to specific loads, because compute and storage can be configured for concret...

show abstract

“…RDMA can be implemented over different link layer protocols. It was originally associated with InfiniBand [29], requiring special-purpose interconnect. Two recent Ethernetbase alternatives -RoCE and iWARP -provide a more cost-efficient solution that is expected to increase RDMA adoption in the data center domain.…”

Section: Nvme-over-fabricsmentioning

confidence: 99%

NVMe-over-fabrics performance characterization and the path to low-overhead flash disaggregation

Guz

Shayesteh

et al. 2017

Proceedings of the 10th ACM International Systems and Storage Conference

View full text Add to dashboard Cite

Storage disaggregation separates compute and storage to different nodes in order to allow for independent resource scaling and thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-based SSD) disaggregation is considered more challenging. This is because SSDs are significantly faster than hard drives, so the latency overheads (due to both network and CPU processing) as well as the extra compute cycles needed for the offloading stack become much more pronounced. In this work we characterize the overheads of NVMe-SSD disaggregation. We show that NVMe-over-Fabrics (NVMf)-a recently-released remote storage protocol specificationreduces the overheads of remote access to a bare minimum, thus greatly increasing the cost-efficiency of Flash disaggregation. Specifically, while recent work showed that SSD storage disaggregation via iSCSI degrades application-level throughput by 20%, we report on negligible performance degradation with NVMf-both when using stress-tests as well as with a more-realistic KV-store workload.

show abstract

High performance RDMA-based MPI implementation over InfiniBand

Cited by 36 publications

References 0 publications

Using RDMA efficiently for key-value services

Using RDMA efficiently for key-value services

Performance Characterization of NVMe-over-Fabrics Storage Disaggregation

NVMe-over-fabrics performance characterization and the path to low-overhead flash disaggregation

Contact Info

Product

Resources

About