Harp: Collective Communication on Hadoop

HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack

“…References in current paper give more details of different applications of this concept. We have applied HPC-ABDS to design high performance run time [1] [12], understand the mapping of applications into the software stack [2], design middleware for data-intensive analytics [3] [13], and to delineate a systematic big data benchmark approach [8] [9]. In this paper, we have aimed at an overall discussion of all layers which can be used in many applications of HPC-ABDS and in fact in studies of just the Commodity big data stack.…”

“…Note recent developments at the programming layer like Apache Hive and Drill, which offer high-layer access models like SQL implemented on scalable NoSQL data systems. The communication layer includes Publish-subscribe technology used in many approaches to streaming data as well the HPC communication technologies (MPI) which are much faster than most default Apache approaches but can be added [12] to some systems like Hadoop whose modern version is modular and allows plug-ins for HPC stalwarts like MPI and sophisticated load balancers. The programming layer includes both the classic batch processing typified by Hadoop and streaming by Storm.…”

Model-centric computation abstractions in machine learning applications

“…It contains 1 million words, each with 10 thousand topics, resulting in a total of 10 billion parameters. The [20]. One is "rtt", which follows Computation Model B (uses "rotateGlobal") in Parallelism Form I and Computation Model D in Form II.…”

Towards an Understanding of Facets and Exemplars of Big Data Applications

“…HPC with MPI suggests that one could integrate categories 11.3 and 11.4 into a single environment. This approach is illustrated by the Harp plug-in for Hadoop which supports both models [27]. We recently added the map-streaming architecture of Table 11.5 and Figure 2b); recall that Table 2 listed 41 streaming applications in the 51 use cases.…”

“…However important data analytics involve full matrix algorithms. For example, recent papers [27,29,30] on a new Multidimensional Scaling method use conjugate gradient solvers with full matrices as opposed to the new sparse conjugate gradient benchmark HPCG being developed for supercomputer (Top500) evaluations [31].…”