Distributed evaluation of subgraph queries using worst-case optimal low-memory dataflows

Abstract

We study the problem of finding and monitoring fixed-size subgraphs in a continually changing large-scale graph. We present the first approach that (i) performs worst-case optimal computation and communication, (ii) maintains a total memory footprint linear in the number of input edges, and (iii) scales down per-worker computation, communication, and memory requirements linearly as the number of workers increases, even on adversarially skewed inputs. Our approach is based on worst-case optimal join algorithms, recast as a data-parallel dataflow computation. We describe the general algorithm and modifications that make it robust to skewed data, prove theoretical bounds on its resource requirements in the massively parallel computing model, and implement and evaluate it on graphs containing as many as 64 billion edges. The underlying algorithm and ideas generalize from finding and monitoring subgraphs to the more general problem of computing and maintaining relational equi-joins over dynamic relations.

A Survey on Distributed Graph Pattern Matching in Massive Graphs

“…Two common algorithms are GenericJoin [57] and LeapFrog TrieJoin [87]. Parallel approaches for finding subgraph isomorphism based on these algorithms were proposed by [2] and [29] respectively. Mhedhbi et al combined both binary joins and the GenericJoin algorithm to evaluate subgraph isomorphism in [56].…”

Efficient and Scalable Graph Pattern Mining on GPUs

“…For a given pattern P , there exist multiple valid matching orders. To choose the best performing matching order, prior works [7,21,22,54,60,74,75,95] have proposed various cost models to predict the performance of matching orders, and choose the one with the highest expected performance. Symmetry order is a partial order enforced among data vertices for symmetry breaking, which removes redundant subgraph enumerations (a.k.a automorphism [26]), and thus guarantees that any match of P in G is found only once.…”

“…2) G 2 Miner consists of a graph loader, a pattern analyzer, a runtime system, a library of CUDA primitives and a code generator. The user is only responsible for specifying the pattern of interest using General CPU GPU Multi-GPU Order Code Gen EmptyHeaded [2] DFS Graphflow [7,56,76] DFS GraphZero [74,75] DFS GraphPi [95] DFS Peregrine [54] DFS Pangolin [25,26] BFS PBE [43,44] BFS G 2 Miner both Table 1: Comparison of state-of-the-art GPM systems, in terms of support for generality of the programming model, hardware platforms (CPU/GPU/multi-GPU), search orders, and code generation.…”

Adopting worst-case optimal joins in relational database systems

“…All graph data sets are in the form of edge relations in which each tuple represents a directed edge between two nodes identified by unsigned 64-bit integers. Like previous work on the subject [3,6,22,42,48], we focus on undirected 3 and 4-clique queries on these graphs as they are a common subpattern in graph workloads [48]. The queries used in our experiments are available online [16].…”

“…Veldhuizen proposed the well-known Leapfrog Triejoin algorithm that is used in the LogicBlox system and can be implemented on top of existing ordered indexes or plain sorted data [13,54,56]. Variants of such join algorithms have been adopted in distributed query processing [4,6,13,35] graph processing [3,6,22,42,59,62], and general-purpose query processing [2,8].…”