Evaluating the Potential of Cray Gemini Interconnect for PGAS Communication Runtime Systems

Abstract

The Cray Gemini Interconnect has been recently introduced as the next generation network for building scalable multi-petascale supercomputers. The Cray XE6 systems, which use the Gemini Interconnect are becoming available with Message Passing Interface (MPI) and Partitioned Global Address Space (PGAS) Models such as as Global Arrays, Unified Parallel C, Co-Array Fortran and Cascade High Performance Language. These PGAS models use one-sided communication runtime systems such as MPI-Remote Memory Access, Aggregate Remote Memory Copy Interface and proprietary communication runtime systems. The primary objective of our work is to study the potential of Cray Gemini Interconnect by designing application specific micro-benchmarks using the DMAPP userspace library. We design micro-benchmarks to study the performance of simple communication primitives and application specific microbenchmarks to understand the behavior of Gemini Interconnect at scale. In our experiments, the Gemini Interconnect can achieve a peak bandwidth of 6911 MB/s and a latency of 1µs for get communication primitive. Scalability tests for atomic memory operations and shift communication operation up to 65536 processes show the efficacy of the Gemini Interconnect.

Extending Summation Precision for Network Reduction Operations

“…Therefore, BigInt does not increase communication time significantly because the additional bits lead to a minor increase since packet headers and fixed (zero-load) delays in the network and NICs dominate for small transfers. Our results are confirmed by past work [21] which showed that the communication delay for messages with payloads containing double-word, quad-word, and double quad-word variables is identical and approximately 12 µs for up to 512 bytes, which is a larger payload size than BigInts (256 bytes).…”

“…This was previously possible only with double-precision variables due to the complex internal structures of arbitrary-precision libraries. Even though the size of a BigInt variable is 2101 bits-33× larger than a 64-bit double-precision variable-we observe insignificant loss in communication delay due to the fixed latency costs and packet overhead bytes in modern large-scale networks [21]. Therefore, BigInts readily apply to network operations and can be combined with past work on local-node computations that uses sorting and recursion or alternative wide fixedpoint representations with dedicated hardware support [22], [23], [15], in order to provide reproducible system-wide operations with no precision loss.…”

Optimizing fine-grained communication in a biomolecular simulation application on Cray XK6

“…Compiler based approaches, such as OMPI [16], similarly Friedley and Lumsdaine describe a compiler approach, producing a 40% improvement, by exploiting onesided communication via transformation of MPI calls [4]. Similar work exploits one-sided communication within the Partitioned Global Address Space (PGAS) languages like Chapel [2], UPC [20], Global Arrays [14], and Co-Array FORTRAN [15] with some preliminary Gemini work using the DMAPP API [21].…”

An Evaluation of One-Sided and Two-Sided Communication Paradigms on Relaxed-Ordering Interconnect

“…Vishnu et al [23], [24] present the implementation of the Aggregate Remote Memory Copy Interface (ARMCI) on Cray XE6 using DMAPP with relaxed ordering. Shan et al [25] present a performance evaluation of UPC and MPI benchmarks on Gemini and show applications using single-sided communication outperform those using twosided paradigm.…”