PowerLyra

Abstract

Natural graphs with skewed distributions raise unique challenges to distributed graph computation and partitioning. Existing graph-parallel systems usually use a “one-size-fits-all” design that uniformly processes all vertices, which either suffer from notable load imbalance and high contention for high-degree vertices (e.g., Pregel and GraphLab) or incur high communication cost and memory consumption even for low-degree vertices (e.g., PowerGraph and GraphX). In this article, we argue that skewed distributions in natural graphs also necessitate differentiated processing on high-degree and low-degree vertices. We then introduce PowerLyra, a new distributed graph processing system that embraces the best of both worlds of existing graph-parallel systems. Specifically, PowerLyra uses centralized computation for low-degree vertices to avoid frequent communications and distributes the computation for high-degree vertices to balance workloads. PowerLyra further provides an efficient hybrid graph partitioning algorithm (i.e., hybrid-cut) that combines edge-cut (for low-degree vertices) and vertex-cut (for high-degree vertices) with heuristics. To improve cache locality of inter-node graph accesses, PowerLyra further provides a locality-conscious data layout optimization. PowerLyra is implemented based on the latest GraphLab and can seamlessly support various graph algorithms running in both synchronous and asynchronous execution modes. A detailed evaluation on three clusters using various graph-analytics and MLDM (Machine Learning and Data Mining) applications shows that PowerLyra outperforms PowerGraph by up to 5.53X (from 1.24X) and 3.26X (from 1.49X) for real-world and synthetic graphs, respectively, and is much faster than other systems like GraphX and Giraph, yet with much less memory consumption. A porting of hybrid-cut to GraphX further confirms the efficiency and generality of PowerLyra.

The Dark Path to Eternal Life: Machiavellianism Predicts Approval of Mind Upload Technology

“…Cut-off points of .01, .05, and .08 have been suggested, corresponding to excellent, good, and mediocre fits respectively (MacCallum, Browne and Sugawara, 1996); confidence intervals should be used to understand the size of sampling error (upper-bound should preferably be < .1). For many of our models presented in this paper, the RMSEA index is only in the range of "good", however this is mostly due to the fact that -irrespective of the size of the data -scales with less than 10 items give inflated RMSEA values, especially if the scale has relatively high factor-loadings (Chen et al, 2019).…”

Trust Management Scheme of IoV Based on Dynamic Sharding Blockchain

“…Static graph-partitioning algorithms require access to the global data of the graph before partitioning, and typically employ spectral methods, heuristic methods, or multi-level partitioning methods to partition the graph data. For instance, the classic algorithm Metis [18] utilizes a multi-level partitioning strategy, while other methods, such as NE [4] and PowerLyra [19], use vertex-cut and hybrid-cut, respectively. By contrast, dynamic graph partitioning, also known as streaming graph-partitioning algorithms, does not require access to the global data of the graph before partitioning.…”

Sancus

“…The distributed GNNs are still in its infancy [2], with a few prior works on GPU-based systems. Compared to distributed systems for large graph analysis [6,10,11,28,32], the communication overhead in distributed GNNs are even more challenging, since the intensive data movement among workers to fetch neighbor embeddings is expensive. Currently, most existing systems utilized a centralized architecture.…”