Parallel processing of filtered queries in attributed semantic graphs

Abstract

Execution of complex analytic queries on massive semantic graphs is a challenging problem in big-data analytics that requires high-performance parallel computing. In a semantic graph, vertices and edges carry attributes of various types and the analytic queries typically depend on the values of these attributes. Thus, the computation must view the graph through a filter that passes only those individual vertices and edges of interest. Previous investigations have developed Knowledge Discovery Toolbox (KDT), a sophisticated a Python library for parallel graph computations. In KDT, the user can write custom graph algorithms by specifying operations between edges and vertices (semiring operations). The user can also customize existing graph algorithms by writing filters. Although the high-level language for this customization enables domain scientists to productively express their graph analytics requirements, the customized queries perform poorly due to the overhead of having to call into the Python virtual machine for each vertex and edge.In this work, we use the Selective Embedded Just-In-Time Specialization (SEJITS) approach to automatically translate semiring operations and filters defined by programmers into a lower-level efficiency language, bypassing the upcall into Python. We evaluate our approach by comparing it with the high-performance Combinatorial BLAS engine and show that our approach combines the benefits of programming in a highlevel language with executing in a low-level parallel environment. We increase the system's flexibility by developing techniques that provide users with the ability to define new vertex and edge types from Python.We also present a new Roofline model for graph traversals and show that we achieve performance that is significantly closer to the bounds suggested by the Roofline. Finally, to further understand the complex interaction with the underlying architecture, we present an analysis using performance counters that quantifies the improvement in hardware behavior in the context our SEJITS methodology. Overall, we demonstrate the first known solution to the problem of obtaining high performance from a productivity language when applying graph algorithms selectively on semantic graphs with hundreds of millions of edges and scaling to thousands of processors for graphs.

Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication

“…The linear algebraic formulation of Luby's randomized MIS algorithm [37] was originally described earlier [38]. Here, we generalize it to distance-2 case, which is shown in Algorithm 3 at a high level.…”

Data mining techniques for IoT analytics

“…2) Parallel processing methods for semi-structural data: for the RDF data processing task, effectiveness and tunable data partitioning framework SPA [32], that use at distributing processing of big RDF data, is presented to fast processing support of different size as well as complexity. A MapReduce framework is designed to carry out SPARQL query processing.…”

“…Data mining techniques for IoT based applications has been widely presented in literature for different decision tasks, such as supervised and unsupervised learning for IoT applications [5,14,32], frequent pattern recognition and association analysis [4,19,21], massive IoT data mining [22,24], stream data mining [36], etc.…”

“…The detailed surveys on the approaches, tools and techniques employed in existing for IoT data mining can be found in [2,23]. Practical approaches in massive data processing for IoT applications are present in [31,32,33] for parallel and distributed data processing.…”

Ideal basis in constructions defined by directed graphs

“…[54]). In graph theory, semirings have been applied, for example, to the investigation of trust networks defined as directed graphs [23], to the study of planar flows of directed graphs [28], and to the design of a sophisticated Python library for parallel graph computations [56].…”