Comparison based sorting for systems with multiple GPUs

Abstract

As a basic building block of many applications, sorting algorithms that efficiently run on modern machines are key for the performance of these applications. With the recent shift to using GPUs for general purpose compuing, researches have proposed several sorting algorithms for single-GPU systems. However, some workstations and HPC systems have multiple GPUs, and applications running on them are designed to use all available GPUs in the system.\ud In this paper we present a high performance multi-GPU merge sort algorithm that solves the problem of sorting data distributed across several GPUs. Our merge sort algorithm first sorts the data on each GPU using an existing single-GPU sorting algorithm. Then, a series of merge steps produce a globally sorted array distributed across all the GPUs in the system. This merge phase is enabled by a novel pivot selection algorithm that ensures that merge steps always distribute data evenly among all GPUs. We also present the implementation of our sorting algorithm in CUDA, as well as a novel inter-GPU communication technique that enables this pivot selection algorithm. Experimental results show that an efficient implementation of our algorithm achieves a speed up of 1.9x when running on two GPUs and 3.3x when running on four GPUs, compared to sorting on a single GPU. At the same time, it is able to sort two and four times more records, compared to sorting on one GPU.Peer ReviewedPostprint (published version

Data Partitioning Strategy of GPU Heterogeneous Clusters Based on Learning

“…Their results show that when compared to a static configuration, they can respond much better to peaks and troughs, achieving up to four times the multi-tenant density on a single server while offering clients the best possible graphics quality. In paper [15], the authors present a high performance multi-GPU merge sort algorithm that solves the problem of sorting data distributed across several GPUs. The merge sort algorithm first sorts the data on each GPU using an existing single-GPU sorting algorithm.…”

GPU-SM: shared memory multi-GPU programming

“…Compute boundaries (lines [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33]: kernels are launched to compute the points in the boundaries, so that they can be communicated as soon as possible. Kernels are queued into different streams so they can be concurrently executed on the GPU.…”

“…While these features have been available for some time, their utilization has been mainly restricted to accelerate bulk data transfers between GPU memories and to enable better integration with I/O devices. Only a few works exploit remote memory accesses in multi-GPU computations [28]. However, we argue that they can be used in many other types of computations.…”

Moim: A Multi-GPU MapReduce Framework

“…As GPU computational capacities of GPUs increase, GPUs are used to support diverse applications, such as sorting [25], text processing [6], Fast Fourier Transform [10], matrix operations [4], databases [1], encryption and decryption [24], intrusion detection [20], and biological applications [18]. These efforts call for a more general parallel programming framework, such as MapReduce, to exploit the parallel computational power provided by CPUs and GPUs with lower complexity compared with manual architectural optimizations.…”