A Tale of Two Data-Intensive Paradigms: Applications, Abstractions, and Architectures

Abstract

Abstract-Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, co-placement and scheduling of data with compute resources, and storing and transferring large volumes of data. We analyze the ecosystems of the two prominent paradigms for data-intensive applications, hereafter referred to as the high-performance computing and the Apache-Hadoop paradigm. We propose a basis, common terminology and functional factors upon which to analyze the two approaches of both paradigms. We discuss the concept of "Big Data Ogres" and their facets as means of understanding and characterizing the most common application workloads found across the two paradigms. We then discuss the salient features of the two paradigms, and compare and contrast the two approaches. Specifically, we examine common implementation/approaches of these paradigms, shed light upon the reasons for their current "architecture" and discuss some typical workloads that utilize them. In spite of the significant software distinctions, we believe there is architectural similarity. We discuss the potential integration of different implementations, across the different levels and components. Our comparison progresses from a fully qualitative examination of the two paradigms, to a semi-quantitative methodology. We use a simple and broadly used Ogre (K-means clustering), characterize its performance on a range of representative platforms, covering several implementations from both paradigms. Our experiments provide an insight into the relative strengths of the two paradigms. We propose that the set of Ogres will serve as a benchmark to evaluate the two paradigms along different dimensions.

Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science

“…It is also worth noting that this workflow is essentially a materials science adaptation of existing similar workflows of data-driven analytics in other domains, as most of the advanced techniques for big data management and informatics come from the field of computer science and more specifically high-performance data mining, [42][43][44][45][46][47][48][49][50] via applications in many different [64][65][66][67][68] and social media analytics, 69-71 among many others.…”

Spark deployment and performance evaluation on the MareNostrum supercomputer

“…In [8], the authors compare the performance of traditional HPC setups against Hadoop-like frameworks over clusters of commodity hardware with respect to the processing of dataintensive workloads. They also propose a set of Big Data applications as a benchmark to investigate and evaluate the different paradigms.…”

Supporting High Performance Molecular Dynamics in Virtualized Clusters using IOMMU, SR-IOV, and GPUDirect

“…Request permissions from permissions@acm.org. duce, among others, have already percolated into data intensive computing within HPC [20]. In addition, there are efforts to support traditional HPC-centric scientific computing applications in virtualized cloud infrastructure.…”