A Flexible Open-Source Toolbox for Scalable Complex Graph Analysis

Lugowski, Adam; Alber, David M.; Buluç, Aydın; Gilbert, John R.; Reinhardt, Steven P.; Teng, Yan; Waranis, Andrew

doi:10.1137/1.9781611972825.80

Cited by 63 publications

(36 citation statements)

References 23 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, some distributed graph processing systems, such as KDT [34], PowerGraph [18], and GPS [42], use graph partitioning techniques that can reduce the communication cost significantly when executing some algorithms on input graphs with skewed degree distributions. In this paper we focus on optimizations that are algorithmic and do not appear to have any system-level equivalents.…”

Section: Name Description Section Strongly Connected Components (Scc)mentioning

confidence: 99%

See 1 more Smart Citation

Optimizing graph algorithms on pregel-like systems

Salihoğlu

Widom

2014

Proc. VLDB Endow.

View full text Add to dashboard Cite

We study the problem of implementing graph algorithms efficiently on Pregel-like systems, which can be surprisingly challenging. Standard graph algorithms in this setting can incur unnecessary inefficiencies such as slow convergence or high communication or computation cost, typically due to structural properties of the input graphs such as large diameters or skew in component sizes. We describe several optimization techniques to address these inefficiencies. Our most general technique is based on the idea of performing some serial computation on a tiny fraction of the input graph, complementing Pregel's vertex-centric parallelism. We base our study on thorough implementations of several fundamental graph algorithms, some of which have, to the best of our knowledge, not been implemented on Pregel-like systems before. The algorithms and optimizations we describe are fully implemented in our open-source Pregel implementation. We present detailed experiments showing that our optimization techniques improve runtime significantly on a variety of very large graph datasets.

show abstract

Section: Name Description Section Strongly Connected Components (Scc)mentioning

confidence: 99%

“…References [34,18,42] describe graph partitioning techniques for assigning graph vertices to machines, with the goal of reducing the communication cost of algorithms. These techniques are effective when vertices send the same message to all of their neighbors in each phase of the algorithm.…”

Section: Related Workmentioning

confidence: 99%

Optimizing graph algorithms on pregel-like systems

Salihoğlu

Widom

2014

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

“…The Knowledge Discovery Toolbox [23] is a flexible Python-based open-source toolbox for implementing complex graph algorithms and executing them on high-performance parallel computers. KDT achieves high performance by invoking linear-algebraic computational primitives supplied by a parallel C++/MPI backend, the Combinatorial BLAS [7].…”

Section: Sejits For Performancementioning

confidence: 99%

“…The Knowledge Discovery Toolbox [23], [24] is a flexible open-source toolkit for complex graph algorithms on highperformance parallel computers. KDT targets two classes of users: domain-expert analysts who are not graph experts, who use KDT by invoking existing routines from Python, and graph-algorithm developers who write Python code that invokes and composes KDT computational primitives.…”

Section: B Kdt Filters In Pythonmentioning

confidence: 99%

High-Productivity and High-Performance Analysis of Filtered Semantic Graphs

Buluç

Duriakova

Fox

et al. 2013

2013 IEEE 27th International Symposium on Parallel and Distributed Processing

Self Cite

View full text Add to dashboard Cite

Abstract-High performance is a crucial consideration when executing a complex analytic query on a massive semantic graph. In a semantic graph, vertices and edges carry attributes of various types. Analytic queries on semantic graphs typically depend on the values of these attributes; thus, the computation must view the graph through a filter that passes only those individual vertices and edges of interest.Knowledge Discovery Toolbox (KDT), a Python library for parallel graph computations, is customizable in two ways. First, the user can write custom graph algorithms by specifying operations between edges and vertices. These programmer-specified operations are called semiring operations due to KDT's underlying linear-algebraic abstractions. Second, the user can customize existing graph algorithms by writing filters that return true for those vertices and edges the user wants to retain during algorithm execution. For high productivity, both semiring operations and filters are written in a high-level language, resulting in relatively low performance due to the bottleneck of having to call into the Python virtual machine for each vertex and edge.In this work, we use the Selective Embedded JIT Specialization (SEJITS) approach to automatically translate semiring operations and filters defined by programmers into a lower-level efficiency language, bypassing the upcall into Python. We evaluate our approach by comparing it with the high-performance Combinatorial BLAS engine, and show our approach enables users to write in high-level languages and still obtain the high performance of low-level code. We also present a new roofline model for graph traversals, and show that our high-performance implementations do not significantly deviate from the roofline. Overall, we demonstrate the first known solution to the problem of obtaining high performance from a productivity language when applying graph algorithms selectively on semantic graphs.

show abstract

“…The Knowledge Discovery Toolbox [32] is a flexible, Python-based, open-source toolbox for implementing complex graph algorithms and executing them on high-performance parallel computers. KDT achieves high performance by invoking linear-algebraic computational primitives supplied by a parallel C++/MPI backend -the Combinatorial BLAS [13].…”

Section: Introductionmentioning

confidence: 99%

Parallel processing of filtered queries in attributed semantic graphs

Lugowski

Kamil

Buluç

et al. 2015

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

Execution of complex analytic queries on massive semantic graphs is a challenging problem in big-data analytics that requires high-performance parallel computing. In a semantic graph, vertices and edges carry attributes of various types and the analytic queries typically depend on the values of these attributes. Thus, the computation must view the graph through a filter that passes only those individual vertices and edges of interest. Previous investigations have developed Knowledge Discovery Toolbox (KDT), a sophisticated a Python library for parallel graph computations. In KDT, the user can write custom graph algorithms by specifying operations between edges and vertices (semiring operations). The user can also customize existing graph algorithms by writing filters. Although the high-level language for this customization enables domain scientists to productively express their graph analytics requirements, the customized queries perform poorly due to the overhead of having to call into the Python virtual machine for each vertex and edge.In this work, we use the Selective Embedded Just-In-Time Specialization (SEJITS) approach to automatically translate semiring operations and filters defined by programmers into a lower-level efficiency language, bypassing the upcall into Python. We evaluate our approach by comparing it with the high-performance Combinatorial BLAS engine and show that our approach combines the benefits of programming in a highlevel language with executing in a low-level parallel environment. We increase the system's flexibility by developing techniques that provide users with the ability to define new vertex and edge types from Python.We also present a new Roofline model for graph traversals and show that we achieve performance that is significantly closer to the bounds suggested by the Roofline. Finally, to further understand the complex interaction with the underlying architecture, we present an analysis using performance counters that quantifies the improvement in hardware behavior in the context our SEJITS methodology. Overall, we demonstrate the first known solution to the problem of obtaining high performance from a productivity language when applying graph algorithms selectively on semantic graphs with hundreds of millions of edges and scaling to thousands of processors for graphs.

show abstract

A Flexible Open-Source Toolbox for Scalable Complex Graph Analysis

Cited by 63 publications

References 23 publications

Optimizing graph algorithms on pregel-like systems

Optimizing graph algorithms on pregel-like systems

High-Productivity and High-Performance Analysis of Filtered Semantic Graphs

Parallel processing of filtered queries in attributed semantic graphs

Contact Info

Product

Resources

About