Can Modern Graph Processing Engines Run Concurrent Queries Efficiently?

Hauck, Matthias; Paradies, Marcus; Fröning, Holger

doi:10.1145/3078447.3078452

Cited by 5 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to compare the estimates for the parallel and the sequential case, we use the total cost per vertex 𝐶 total (𝑇 , 𝑀). This cost is the sum of the costs to process the vertex itself and the cost of its share of edges and the newly found vertices, as shown in Equation (8).…”

Section: Cost Modelmentioning

confidence: 99%

“…For multi-query execution, in the following referred to as number of concurrent queries, the problem worsens as an additional dimension is added, which is to select a schedule with the right amount and set of queries to be processed concurrently. In one of our previous works we showed that multi-query execution using simply multiple parallel engines at the same time can degrade performance [8]. To overcome this, given enough concurrent queries ready for execution, a promising option might be to process each query internally sequential (intra-query), what minimizes synchronization overhead, and thereby sourcing the required degree of parallelism from many concurrent queries (inter-query).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scheduling of Graph Queries: Controlling Intra- and Inter-query Parallelism for a High System Throughput

Hauck¹,

Oukid²,

Fröning³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The vast amounts of data used in social, business or traffic networks, biology and other natural sciences are often managed in graph-based data sets, consisting of a few thousand up to billions and trillions of vertices and edges, respectively. Typical applications utilizing such data either execute one or a few complex queries or many small queries at the same time interactively or as batch jobs. Furthermore, graph processing is inherently complex, as data sets can substantially differ (scale free vs. constant degree), and algorithms exhibit diverse behavior (computational intensity, local or global, push-or pull-based).This work is concerned with multi-query execution by automatically controlling the degree of parallelization, with overall objectives including high system utilization, low synchronization cost, and highly efficient concurrent execution. The underlying concept is three-fold: (1) sampling is used to determine graph statistics, (2) parallelization constraints are derived from algorithm and system properties, and (3) suitable work packages are generated based on the previous two aspects. We evaluate the proposed concept using different algorithms on synthetic and real world data sets, with up to 16 concurrent sessions (queries). The results demonstrate a robust performance in spite of these various configurations, and in particular that the performance is always close to or even slightly ahead of the performance of manually optimized implementations. Furthermore, the similar performance to manually optimized implementations under extreme configurations, which require either a full parallelization (few large queries) or complete sequential execution (many small queries), shows that the proposed concept exhibits a particularly low overhead.

show abstract

Section: Cost Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Scheduling of Graph Queries: Controlling Intra- and Inter-query Parallelism for a High System Throughput

Hauck¹,

Oukid²,

Fröning³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…While CGraph processes FPP queries by iterations, ForkGraph processes FPP queries in the partitionlevel granularity, leveraging work-efficient sequential executions. Hauck et al [29] motivate the experimental studies of processing concurrent queries based on the multi-user setups in classic relational enterprise database environments or web-scale environments. They study the inter-and intra-parallelism of handling different types of graph queries by assigning different threads to different instances of Galios [45].…”

Section: Related Workmentioning

confidence: 99%

Cache-Efficient Fork-Processing Patterns on Large Graphs

Lu,

Sun,

Paul

et al. 2021

Preprint

View full text Add to dashboard Cite

As large graph processing emerges, we observe a costly fork-processing pattern (FPP) common in many graph algorithms. The unique feature of the FPP is that it launches many independent queries from different source vertices on the same graph. For example, an algorithm in analyzing the network community profile can execute Personalized PageRanks that start from tens of thousands of source vertices at the same time. We study the efficiency of state-of-the-art graph processing systems on multi-core architectures, including Ligra, Gemini, and GraphIt. We find that those systems suffer from severe cache miss penalty because of the irregular and uncoordinated memory accesses in processing FPPs.In this paper, we propose ForkGraph, a cache-efficient FPP processing system on multi-core architectures. In order to improve the cache reuse, we divide the graph into partitions each sized of LLC (last-level cache) capacity, and the queries in an FPP are buffered and executed on the partition basis. We further develop efficient intra-and inter-partition execution strategies for efficiency. For intra-partition processing, since the graph partition fits into LLC, we propose to execute each graph query with efficient sequential algorithms (in contrast with parallel algorithms in existing parallel graph processing systems) and present an atomic-free query processing by consolidating contending operations to cache-resident graph partition. For inter-partition processing, we propose two designs, yielding and priority-based scheduling, to reduce redundant work in processing. Besides, we theoretically prove that ForkGraph performs the same amount of work, to within a constant factor, as the fastest known sequential algorithms in FPP queries processing, which is work efficient. Our evaluations on real-world graphs show that ForkGraph significantly outperforms state-of-the-art graph processing systems (including Ligra, Gemini, and GraphIt) with two orders of magnitude speedups.

show abstract

“…All of the presented approaches support only a single, batch processing query rather than multiple, interactive graph queries. Hauck et al [14] provided experimental evidence that single-query graph systems do not scale well in a multi-query environment.…”

Section: Related Workmentioning

confidence: 99%

Q-graph

Mayer

Grunert

et al. 2018

Proceedings of the 1st ACM SIGMOD Joint International Workshop on Graph Data Management Experiences &Amp; Systems (GRADES) and

View full text Add to dashboard Cite

Arising user-centric graph applications such as route planning and personalized social network analysis have initiated a shift of paradigms in modern graph processing systems towards multiquery analysis, i.e., processing multiple graph queries in parallel on a shared graph. These applications generate a dynamic number of localized queries around query hotspots such as popular urban areas. However, existing graph processing systems are not yet tailored towards these properties: The employed methods for graph partitioning and synchronization management disregard query locality and dynamism which leads to high query latency. To this end, we propose the system Q-Graph for multi-query graph analysis that considers query locality on three levels. (i) The query-aware graph partitioning algorithm Q-cut maximizes query locality to reduce communication overhead. (ii) The method for synchronization management, called hybrid barrier synchronization, allows for full exploitation of local queries spanning only a subset of partitions. (iii) Both methods adapt at runtime to changing query workloads in order to maintain and exploit locality. Our experiments show that Q-cut reduces average query latency by up to 57 percent compared to static query-agnostic partitioning algorithms.

show abstract

Can Modern Graph Processing Engines Run Concurrent Queries Efficiently?

Cited by 5 publications

References 10 publications

Scheduling of Graph Queries: Controlling Intra- and Inter-query Parallelism for a High System Throughput

Scheduling of Graph Queries: Controlling Intra- and Inter-query Parallelism for a High System Throughput

Cache-Efficient Fork-Processing Patterns on Large Graphs

Q-graph

Contact Info

Product

Resources

About