Comparison of sort algorithms in Hadoop and PCJ

Abstract

Sorting algorithms are among the most commonly used algorithms in computer science and modern software. Having efficient implementation of sorting is necessary for a wide spectrum of scientific applications. This paper describes the sorting algorithm written using the partitioned global address space (PGAS) model, implemented using the Parallel Computing in Java (PCJ) library. The iterative implementation description is used to outline the possible performance issues and provide means to resolve them. The key idea of the implementation is to have an efficient building block that can be easily integrated into many application codes. This paper also presents the performance comparison of the PCJ implementation with the MapReduce approach, using Apache Hadoop TeraSort implementation. The comparison serves to show that the performance of the implementation is good enough, as the PCJ implementation shows similar efficiency to the Hadoop implementation.

Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)

“…In this implementation only the single SealedObject is created for each PCJ thread and the object is reused for all iterations of the main loop. Full source code is available on GitHub 49 …”

“…37,38 However, one could argue, that the implemented problem was not well-suited for Hadoop processing. The paper 39 presents the TeraSort benchmark used for measuring the performance of Hadoop clusters. For that problem, the PCJ implementation shows better or at least the same performance as the Apache Hadoop implementation, even with the data placement agnostic approach for reading data from the HDFS system.…”