Towards a Collective Layer in the Big Data Stack

Gunarathne, Thilina; Qiu, Judy; Gannon, Dennis

doi:10.1109/ccgrid.2014.123

Cited by 6 publications

(2 citation statements)

References 14 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As such, research regarding virtualization can also play a part in bringing advanced hardware and performance-focused considerations to Big Data applications, effectively cross-cutting the convergence with HPC. Recent efforts have taken place utilizing collectives found in HPC applications within big data frameworks [17], leveraging high-speed, low-latency interconnects directly in Map Reduce frameworks like Hadoop [21], and investigating areas where performance-centric lessons learned can be leveraged within big data stacks [2]. These endeavors are collectively pushing forward the notion of cross-cutting convergence within analytics platform services themselves.…”

Section: Data Analytics On Cloud Infrastructurementioning

confidence: 99%

Enabling Diverse Software Stacks on Supercomputers Using High Performance Virtual Clusters

Younge

Pedretti

Grant

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

While large-scale simulations have been the hallmark of the High Performance Computing (HPC) community for decades, Large Scale Data Analytics (LSDA) workloads are gaining attention within the scientific community not only as a processing component to large HPC simulations, but also as standalone scientific tools for knowledge discovery. With the path towards Exascale, new HPC runtime systems are also emerging in a way that differs from classical distributed computing models. However, system software for such capabilities on the latest extreme-scale DOE supercomputing needs to be enhanced to more appropriately support these types of emerging software ecosystems.

show abstract

Section: Data Analytics On Cloud Infrastructurementioning

confidence: 99%

Enabling Diverse Software Stacks on Supercomputers Using High Performance Virtual Clusters

Younge

Pedretti

Grant

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

show abstract

“…Extending collective communication to commodity clusters. These systems [9,11] alleviate the effect of degreeskew on small-scale clusters, but the impact of concurrent network transactions can be significant for collective communication on large-scale clusters, and thus degrades the performance of these systems in processing large graphs.…”

Section: Introductionmentioning

confidence: 99%

Towards Scale-out Capability on Social Graphs

Shang

Zhao

Kiran

2015

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

The development of cloud storage and computing has facilitated the rise of various big data applications. As a representative high performance computing (HPC) workload, graph processing is becoming a part of cloud computing. However, scalable computing on large graphs is still dominated by HPC solutions, which require high performance all-to-all collective operations over torus (or mesh) networking. Implementing those torus-based algorithms on commodity clusters, e.g., cloud computing infrastructures, can result in great latency due to inefficient communication. Moreover, designing a highly scalable system for large social graphs, is far from being trivial, as intrinsic features of social graphs, e.g., degree skewness and lacking of locality, often profoundly limit the extent of parallelism. To resolve the challenges, we explore the iceberg of developing a scalable system for processing large social graphs on commodity clusters. In particular, we focus on the scale-out capability of the system. We propose a novel separatorcombiner based query processing engine which provides native load-balancing and very low communication overhead, such that increasinglylarger graphs can be simply addressed by adding more computing nodes to the cluster.The proposed system achieves remarkable scale-out capability in processing large social graphs with skew degree distributions, while providing many critical features for big data analytics, such as easy-to-use API, fault-tolerance and recovery. We implement the system as a portable and easily configurable library, and conduct comprehensive experimental studies to demonstrate its effectiveness and efficiency.

show abstract