A cloud-based approach to big graphs

Burkhardt, Paul; Waring, Christopher A.

doi:10.1109/hpec.2015.7396313

Cited by 9 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most users deploy Accumulo on a sizable cluster. Graphs as large as 70 trillion edges (1.1 PB) have been analyzed by clusters as large as 1200 machines and 57.6 TB collective memory [25]. This work evaluates single-node performance as a proof of concept and an indicator for how multi-node performance might scale.…”

Section: Performancementioning

confidence: 99%

From NoSQL Accumulo to NewSQL Graphulo: Design and utility of graph algorithms inside a BigTable database

Hutchison

Kepner

Gadepally

et al. 2016

2016 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

Abstract-Google BigTable's scale-out design for distributed key-value storage inspired a generation of NoSQL databases. Recently the NewSQL paradigm emerged in response to analytic workloads that demand distributed computation local to data storage. Many such analytics take the form of graph algorithms, a trend that motivated the GraphBLAS initiative to standardize a set of matrix math kernels for building graph algorithms. In this article we show how it is possible to implement the GraphBLAS kernels in a BigTable database by presenting the design of Graphulo, a library for executing graph algorithms inside the Apache Accumulo database. We detail the Graphulo implementation of two graph algorithms and conduct experiments comparing their performance to two main-memory matrix math systems. Our results shed insight into the conditions that determine when executing a graph algorithm is faster inside a database versus an external system-in short, that memory requirements and relative I/O are critical factors.

show abstract

Section: Performancementioning

confidence: 99%

From NoSQL Accumulo to NewSQL Graphulo: Design and utility of graph algorithms inside a BigTable database

Hutchison

Kepner

Gadepally

et al. 2016

2016 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

show abstract

“…The Graph500 generator is scalable, can be run in parallel without requiring communication between processors, and has been used to generate some of the largest graphs in the world [Burkhardt & Waring 2015, Kepner et al 2014. Each edge in the graph is defined by a pair of numbers representing the start and end vertices of the edge.…”

Section: A Kernel 0: Generate Graphmentioning

confidence: 99%

PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms

Dreher¹,

Byun²,

Hill³

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

View full text Add to dashboard Cite

The rise of big data systems has created a need for benchmarks to measure and compare the capabilities of these systems. Big data benchmarks present unique scalability challenges. The supercomputing community has wrestled with these challenges for decades and developed methodologies for creating rigorous scalable benchmarks (e.g., HPC Challenge). The proposed PageRank pipeline benchmark employs supercomputing benchmarking methodologies to create a scalable benchmark that is reflective of many real-world big data processing systems. The PageRank pipeline benchmark builds on existing prior scalable benchmarks (Graph500, Sort, and PageRank) to create a holistic benchmark with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. The linear algebraic nature of PageRank makes it well suited to being implemented using the GraphBLAS standard.The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed PageRank pipeline benchmark is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance has been measured.

show abstract

“…We refer to such analytics as cued analytics, that is, analytics on selected table subsets. Such analytics are ideally suited for databases by quickly accessing subsets of interest (for example, particular communities in a large social media graph) in contrast to whole-table analytics that may be better suited for parallel processing frameworks such as Hadoop Map-Reduce [19].…”

Section: Experiment: Subgraph Extractionmentioning

confidence: 99%

Benchmarking the graphulo processing framework

Weale

Gadepally

Hutchison

et al. 2016

2016 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

Abstract-Graph algorithms have wide applicablity to a variety of domains and are often used on massive datasets. Recent standardization efforts such as the GraphBLAS specify a set of key computational kernels that hardware and software developers can adhere to. Graphulo is a processing framework that enables GraphBLAS kernels in the Apache Accumulo database. In our previous work, we have demonstrated a core Graphulo operation called TableMult that performs large-scale multiplication operations of database tables. In this article, we present the results of scaling the Graphulo engine to larger problems and scalablity when a greater number of resources is used. Specifically, we present two experiments that demonstrate Graphulo scaling performance is linear with the number of available resources. The first experiment demonstrates cluster processing rates through Graphulo's TableMult operator on two large graphs, scaled between 2 17 and 2 19 vertices. The second experiment uses TableMult to extract a random set of rows from a large graph (2 19 nodes) to simulate a cued graph analytic. These benchmarking results are of relevance to Graphulo users who wish to apply Graphulo to their graph problems.

show abstract

A cloud-based approach to big graphs

Cited by 9 publications

References 20 publications

From NoSQL Accumulo to NewSQL Graphulo: Design and utility of graph algorithms inside a BigTable database

From NoSQL Accumulo to NewSQL Graphulo: Design and utility of graph algorithms inside a BigTable database

PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms

Benchmarking the graphulo processing framework

Contact Info

Product

Resources

About