Comparison of the HPC and Big Data Java Libraries Spark, PCJ and APGAS

Posner, Jonas; Reitz, Lukas; Fohry, Claudia

doi:10.1109/paw-atm.2018.00007

Cited by 4 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies [40,41] show that the PCJ implementation of some benchmarks outperforms the Hadoop implementation, even by a factor of 100. Other studies that compare PCJ with APGAS and Apache Spark [56] show that the PCJ implementation has the same performance as Apache Spark, and for some benchmarks it can be almost 2-times more efficient. However, not all of the studies were perfectly suited for MapReduce processing.…”

Section: Discussionmentioning

confidence: 93%

Comparison of sort algorithms in Hadoop and PCJ

Nowicki

2020

J Big Data

View full text Add to dashboard Cite

Sorting algorithms are among the most commonly used algorithms in computer science and modern software. Having efficient implementation of sorting is necessary for a wide spectrum of scientific applications. This paper describes the sorting algorithm written using the partitioned global address space (PGAS) model, implemented using the Parallel Computing in Java (PCJ) library. The iterative implementation description is used to outline the possible performance issues and provide means to resolve them. The key idea of the implementation is to have an efficient building block that can be easily integrated into many application codes. This paper also presents the performance comparison of the PCJ implementation with the MapReduce approach, using Apache Hadoop TeraSort implementation. The comparison serves to show that the performance of the implementation is good enough, as the PCJ implementation shows similar efficiency to the Hadoop implementation.

show abstract

Section: Discussionmentioning

confidence: 93%

Comparison of sort algorithms in Hadoop and PCJ

Nowicki

2020

J Big Data

View full text Add to dashboard Cite

show abstract

“…As a result, a higher performance is achieved; unfortunately, it is still lower than for other Java-based solutions. In particular, it has been demonstrated that the performance of applications built using the PCJ library or APGAS (a branch of IBM's X10 language project) is higher than for Hadoop/Spark implementations [10,11]. For the calculation-intensive and communication-intensive HPC workloads, PCJ is several up to hundreds of times faster [12].…”

Section: Map-reducementioning

confidence: 99%

“…We have traditionally used a text of the real book for the word-counting code performance analyses (with other authors already following suit [11]). Georges de Scudéry's Artamène ou le Grand Cyrus [32] is one of the largest novels ever written, and its textual representation takes 10 MB of disk space.…”

Section: Pos(isgc2021)007mentioning

confidence: 99%

Scalable computing in Java with PCJ Library. Improved collective operations

Nowicki¹,

Górski²,

Bała³

2021

Proceedings of International Symposium on Grids &Amp; Clouds 2021 — PoS(ISGC2021)

View full text Add to dashboard Cite

Machine learning and Big Data workloads are becoming as important as traditional HPC ones. AI and Big Data users tend to use new programming languages such as Python, Julia, or Java, while the HPC community is still dominated by C/C++ or Fortran. Hence, there is a need for new programming libraries and languages that will integrate different applications and allow them to run on large computer infrastructure. Since modest computers are multinode and multicore, parallel execution is an additional challenge here. For that purpose, we have developed the PCJ library, which introduces parallel programming capabilities to Java using the Partitioned Global Address Space model. It does not modify language nor running environment (JVM). The PCJ library allows for easy development of parallel code and runs it on laptops, workstations, supercomputers, and the cloud. This paper presents an overview of the PCJ library and its usage in parallelizing selected workloads, including HPC, AI, and Big Data. The performance and scalability are presented. We present recent addition to the PCJ library, which are collective operations. The collective operations significantly reduce the number of lines of code to write, ensuring good performance.

show abstract

“…The performance comparison with the C/MPI based codes has been presented in previous papers 4,5 . The extensive comparison with Java‐based solutions including APGAS (Java implementation of X10 language) has been also performed 3,6 …”

Section: Introductionmentioning

confidence: 99%

“…4,5 The extensive comparison with Java-based solutions including APGAS (Java implementation of X10 language) has been also performed. 3,6 In this article, we contribute by focusing on the performance of selected applications run on Amazon web services (AWS) elastic compute cloud (EC2) using both Intel and ARM architectures, and Linaro Developer Cloud that provides instances with ARM processors. The ultimate aim is to show the feasibility of using Java and the PCJ library for the development of parallel applications run on the cloud.…”

mentioning

confidence: 99%

Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)

Nowicki

Górski

Bała

2021

Concurrency and Computation

View full text Add to dashboard Cite

Large-scale computing and data processing with cloud resources is gaining popularity.However, the usage of the cloud differs from traditional high-performance computing (HPC) systems and both algorithms and codes have to be adjusted. This work is often time-consuming and performance is not guaranteed. To address this problem we have developed the PCJ library (parallel computing in Java), a novel tool for scalable HPC and big data processing in Java. In this article, we present a performance evaluation of parallel applications implemented in Java using the PCJ library. The performance evaluation is based on the examples of highly scalable applications of different characteristics focusing on CPU, communication or I/O. They run on the traditional HPC system and Amazon web services Cloud as well as Linaro Developer Cloud. For the clouds, we have used Intel x86 and ARM processors for running Java codes without changing any line of the program code and without the need for time-consuming recompilation. Presented applications have been parallelized using the partitioned global address space programming model and its realization in the PCJ library. Our results prove that the PCJ library, due to its performance and ability to create simple portable code, has great promise to be successful for the parallelization of various applications and run them on the cloud with a performance close to HPC systems.

show abstract

Comparison of the HPC and Big Data Java Libraries Spark, PCJ and APGAS

Cited by 4 publications

References 14 publications

Comparison of sort algorithms in Hadoop and PCJ

Comparison of sort algorithms in Hadoop and PCJ

Scalable computing in Java with PCJ Library. Improved collective operations

Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)

Contact Info

Product

Resources

About