Distributed GraphLab

Abstract

While high-level data parallel frameworks, like MapReduce, simplify the design and implementation of large-scale data processing systems, they do not naturally or efficiently support many important data mining and machine learning algorithms and can lead to inefficient learning systems. To help fill this critical void, we introduced the GraphLab abstraction which naturally expresses asynchronous, dynamic, graph-parallel computation while ensuring data consistency and achieving a high degree of parallel performance in the shared-memory setting. In this paper, we extend the GraphLab framework to the substantially more challenging distributed setting while preserving strong data consistency guarantees. We develop graph based extensions to pipelined locking and data versioning to reduce network congestion and mitigate the effect of network latency. We also introduce fault tolerance to the GraphLab abstraction using the classic Chandy-Lamport snapshot algorithm and demonstrate how it can be easily implemented by exploiting the GraphLab abstraction itself. Finally, we evaluate our distributed implementation of the GraphLab abstraction on a large Amazon EC2 deployment and show 1-2 orders of magnitude performance gains over Hadoop-based implementations.

Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science

“…It is also worth noting that this workflow is essentially a materials science adaptation of existing similar workflows of data-driven analytics in other domains, as most of the advanced techniques for big data management and informatics come from the field of computer science and more specifically high-performance data mining, [42][43][44][45][46][47][48][49][50] via applications in many different [64][65][66][67][68] and social media analytics, 69-71 among many others.…”

A survey of cloud-based network intrusion detection analysis

“…However, writing new or customizing existing algorithms is too costly because all algorithms need to be implemented (or modified) by following fixed distributed runtime plans and underlying data-parallel framework. To address this limitation, researchers proposed several approaches of fast implementing approaches as SystemML [10], NIMBLE [11], MLbase [12], Distributed GraphLab [13], and Tupleware [14]. These approaches are classified as "Declarative ML" [54].…”

“…Therefore, many popular computational algorithms cannot be directly utilized in cloud computing architecture [8]. To address this limitation, researchers have started designing scalable performant machine learning applications that run in the cloud computing environment [9][10][11][12][13][14]. In addition, many researchers emphasized the importance of machine learning algorithms (MLAs) to intrusion detection analysis using cloud computing technology [15][16][17][18].…”

Spec2Fab

“…It is inspired by probabilistic graphical models [Jordan et al 1999] that have been popular for representing variable dependence in multi-variate optimization problems. These problems can be parallelized [Low et al 2010] according to the associated graph structure.…”