Scalable PGAS collective operations in NUMA clusters

Abstract

The increasing number of cores per processor is turning manycore-based systems in pervasive. This involves dealing with multiple levels of memory in NUMA (Non Uniform Memory Access) systems and processor cores hierarchies, accessible via complex interconnects in order to dispatch the increasing amount of data required by the processing elements. The key for efficient and scalable provision of data is the use of collective communication operations that minimize the impact of bottlenecks. Leveraging one sided communications becomes more important in these systems, to avoid unnecessary synchronization between pairs of processes in collective operations implemented in terms of two sided point to point functions. This work proposes a series of algorithms that provide a good performance and scalability in collective operations, based on the use of hierarchical trees, overlapping one-sided communications, message pipelining and the available NUMA binding features. An implementation has been developed for Unified Parallel C (UPC), a PGAS (Partitioned Global Address Space) language, which presents a shared memory view across the nodes for programmability, while keeping private memory regions for performance. The performance evaluation of the proposed implementation, conducted on five representative systems (JuRoPA, JUDGE, Finis Terrae, SVG and Superdome), has shown generally good performance and scalability, even outperforming MPI in some cases, which confirms the suitability of the developed algorithms for manycore architectures.

MPI and UPC broadcast, scatter and gather algorithms in Xeon Phi

“…This increases the overhead, that becomes more apparent with a large number of threads. Similar algorithms to the ones here presented were studied in the past, in a traditional cluster, with performance advantages over MPI . In Xeon Phi, it looks like our collectives do not achieve a good performance when compared with MPI in certain cases, like on broadcast.…”

“…The correctness of all the variations has been assessed with GWU Unified Testing Suite . These algorithms are heavily based in our previous work , where they were explained and discussed in more detail.…”

“…The correctness of all the variations has been assessed with GWU Unified Testing Suite [37]. These algorithms are heavily based in our previous work [38], where they were explained and discussed in more detail. Generally it is recommended to rely on hybrid parallelisation with few communicating threads or processes to take full advantage of the Xeon Phi computing power in an application.…”