A stream partitioning approach to processing large scale distributed graph datasets

Wang, Rui; Chiu, Kenneth

doi:10.1109/bigdata.2013.6691619

Cited by 9 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From this clear evidence, it follows that scalability issues become a critical challenge to be faced-off, as analytics run over increasing-in-size graphs, even characterized by high-dimensionality. A possible solution to this problem is represented by the usage of parallel processing methodology (e.g., [27,28]), which ensure a critical gain over classical computational approaches.…”

Section: Scalability Issuesmentioning

confidence: 99%

“…Partitioning schemes for processing big graphs efficiently are of critical relevance, as data partitioning seems to be the most effective approach to process big graphs as to deal with their size and dimensionality (e.g., [28]). The nature of graphs makes it difficult to partition them effectively, essentially due to their skewed degree distributions (e.g., [36]).…”

Section: Partitioning Schemes For Processing Big Graphs Efficientlymentioning

confidence: 99%

“…Here, Cloud-enabled databases (e.g., [25,26]) interface heterogeneous big data sources such as flat files, sensor and stream networks, graph databases, Web databases, and so forth, being located in a target Cloud environment. The data-intensive procedures executed within this Cloud environment generate a large-scale big graph, which may be still distributed (e.g., [27,28]) and represents a rich source of knowledge and patterns to be extracted via ad-hoc big graph analytics tools (e.g., [2]) exploited by end-users Alice and Bob. Alice and Bob edit their proper dataintensive analytical workflow (e.g., [29]), which, again, may be modeled in terms of graphs too, which, executed on top of the Cloud environment, produces the final analytics provided via suitable graphical interfaces (e.g., [30]).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Big Graph Analytics

Cuzzocrea

Song

2014

Proceedings of the 17th International Workshop on Data Warehousing and OLAP

View full text Add to dashboard Cite

Analytics over big graphs is becoming a first-class challenge in database research, with fast-growing interest from both the academia and the industrial community. This problem arises in several application scenarios, ranging from social networks to largescale network systems, from knowledge discovery to cybersecurity, and so forth. Following this major trend, this paper explores actual state-of-the-art results in the area of analytics over big graphs and discusses open research issues and actual trends in such area.

show abstract

Section: Scalability Issuesmentioning

confidence: 99%

Section: Partitioning Schemes For Processing Big Graphs Efficientlymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Big Graph Analytics

Cuzzocrea

Song

2014

Proceedings of the 17th International Workshop on Data Warehousing and OLAP

View full text Add to dashboard Cite

show abstract

“…The performance of DisGCo-generated programs depends a lot on how well the graph is loadbalanced among the executing processes. For better load balancing, the input graph has to be partitioned effectively among processes, and efficient partitioning schemes are explored by many prior works in the literature Abdolrashidi and Ramaswamy [5], Ahmed et al [6], Andreev and Räcke [8], Bader and Madduri [9], Nishimura and Ugander [41], Tsourakakis et al [53], Wang and Chiu [54]. There are also standalone tools that partition the graphs for later use [29,50].…”

Section: Graph Partitioningmentioning

confidence: 99%

DisGCo

Rajendran

Nandivada

2020

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

Graph algorithms are widely used in various applications. Their programmability and performance have garnered a lot of interest among the researchers. Being able to run these graph analytics programs on distributed systems is an important requirement. Green-Marl is a popular Domain Specific Language (DSL) for coding graph algorithms and is known for its simplicity. However, the existing Green-Marl compiler for distributed systems (Green-Marl to Pregel) can only compile limited types of Green-Marl programs (in Pregel canonical form). This severely restricts the types of parallel Green-Marl programs that can be executed on distributed systems. We present DisGCo, the first compiler to translate any general Green-Marl program to equivalent MPI program that can run on distributed systems. Translating Green-Marl programs to MPI (SPMD/MPMD style of computation, distributed memory) presents many other exciting challenges, besides the issues related to differences in syntax, as Green-Marl gives the programmer a unified view of the whole memory and allows the parallel and serial code to be intermixed. We first present the set of challenges involved in translating Green-Marl programs to MPI and then present a systematic approach to do the translation. We also present a few optimization techniques to improve the performance of our generated programs. DisGCo is the first graph DSL compiler that can handle all syntactic capabilities of a practical graph DSL like Green-Marl and generate code that can run on distributed systems. Our preliminary evaluation of DisGCo shows that our generated programs are scalable. Further, compared to the state-of-the-art DH-Falcon compiler that translates a subset of Falcon programs to MPI, our generated codes exhibit a geomean speedup of 17.32×.

show abstract

“…For example, [8] adopts hash partitioning on triples subjects using MapReduce. [9] applies a graph partitioning approach for streaming RDF data. Query driven partitioning [3] leverages query knowledge to partition data so as to answer queries by single node computations.…”

Section: Introductionmentioning

confidence: 99%

Incremental Data Partitioning of RDF Data in SPARK

Agathangelos

Troullinou

Kondylakis

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Significant efforts have been dedicated recently to the development of architectures for storing and querying RDF data in distributed environments. Several approaches focus on data partitioning, which are able to answer queries efficiently, by using a small number of computational nodes. However, such approaches provide static data partitions. Given the increase on the continuous and rapid flow of data, nowadays there is a clear need to deal with streaming data. In this work, we propose a framework for incremental data partitioning by exploiting machine learning techniques. Specifically, we present a method to learn the structure of a partitioned database, and we employ two machine learning algorithms, namely Logistic Regression and Random Forest, to classify new streaming data.

show abstract

A stream partitioning approach to processing large scale distributed graph datasets

Cited by 9 publications

References 13 publications

Big Graph Analytics

Big Graph Analytics

DisGCo

Incremental Data Partitioning of RDF Data in SPARK

Contact Info

Product

Resources

About