Evaluating use of data flow systems for large graph analysis

Yoo, Andy; Kaplan, I L

doi:10.1145/1646468.1646473

Cited by 9 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, new complete languages such as X10 [29], ECL [33], UPC [21], Legion [3], and Chapel [4] have been defined by exploiting in them a data-centric approach. Furthermore, new APIs based on a revolutionary approach, such as GA [20] and SHMEM [19], have been implemented according to a library-based model.…”

Section: Exascale Programming Systemsmentioning

confidence: 99%

A view of programming scalable data analysis: from clouds to exascale

Talia

2019

J Cloud Comp

View full text Add to dashboard Cite

Scalability is a key feature for big data analysis and machine learning frameworks and for applications that need to analyze very large and real-time data available from data repositories, social media, sensor networks, smartphones, and the Web. Scalable big data analysis today can be achieved by parallel implementations that are able to exploit the computing and storage facilities of high performance computing (HPC) systems and clouds, whereas in the near future Exascale systems will be used to implement extreme-scale data analysis. Here is discussed how clouds currently support the development of scalable data mining solutions and are outlined and examined the main challenges to be addressed and solved for implementing innovative data analysis applications on Exascale systems.

show abstract

Section: Exascale Programming Systemsmentioning

confidence: 99%

A view of programming scalable data analysis: from clouds to exascale

Talia

2019

J Cloud Comp

View full text Add to dashboard Cite

show abstract

“…Our work shares with Pregel the ideal that graph processing middlewares should: (i) be useable on today's clusters or cloud computing facilities -what excludes works using massively parallel shared memory architectures [15,14] or specialized systems such as IBM's BlueGene/L [21]; (ii) provide a convenient and easy abstraction for defining a graph processing application -what excludes approaches such as using MapReduce for graph processing [3] or other existing distributed processing frameworks [7] or databases [22]; (iii) be able to distribute the cost of storing and executing an algorithm in a large graph -what excludes single-node approaches for graph processing simply by the fact that they do not provide enough primary or even secondary memory for storing and retrieving said graphs [14].…”

Section: Related Workmentioning

confidence: 99%

A middleware for parallel processing of large graphs

Macambira

Guedes

2010

Proceedings of the 8th International Workshop on Middleware for Grids, Clouds and E-Science

View full text Add to dashboard Cite

With the increasing "data deluge" scientists face today, the analysis and processing of large datasets of structured data is a daring task. Among such data, large graphs are gaining particular importance with the growing interest on social networks and other complex networks. Given the dimensions considered, parallel processing is essential. However, users are generally not experts in writing parallel code to handle such structures. In this work we present Rendero, a middleware that makes it possible to easily describe graph algorithms in a form adequate for parallel execution. The system is based on the Bulk-Synchronous programming model and offers a vertex-based abstraction. Our current implementation offers good speed-up and scale-up results for large graphs ranging from tens of thousands to millions of vertices and edges in some cases.

show abstract

“…LexisNexis performed at levels in between the SQL and Hadoop systems and offered the most flexible means of specifying dataflow computations. Other researchers have reported that LexisNexis offers exceptional speedups in graph analysis algorithms [19]. Finally, Hadoop clusters can deliver excellent performance in brute-force applications, provide high-levels of fault tolerance, and are highly accessible to researchers.…”

Section: Chapter 8 Conclusionmentioning

confidence: 99%

Scientific data analysis on data-parallel platforms.

Ulmer¹,

Bayer²,

Choe³

et al. 2010

View full text Add to dashboard Cite

As scientific computing users migrate to petaflop platforms that promise to generate multiterabyte datasets, there is a growing need in the community to be able to embed sophisticated analysis algorithms in the computing platforms' storage systems. Data Warehouse Appliances (DWAs) are attractive for this work, due to their ability to store and process massive datasets efficiently. While DWAs have been utilized effectively in data-mining and informatics applications, they remain largely unproven in scientific workloads. In this paper we present our experiences in adapting two mesh analysis algorithms to function on five different DWA architectures: two Netezza database appliances, an XtremeData dbX database, a LexisNexis DAS, and multiple Hadoop MapReduce clusters. The main contribution of this work is insight into the differences between these DWAs from a user's perspective. In addition, we present performance measurements for ten DWA systems to help understand the impact of different architectural trade-offs in these systems.

show abstract

Evaluating use of data flow systems for large graph analysis

Cited by 9 publications

References 17 publications

A view of programming scalable data analysis: from clouds to exascale

A view of programming scalable data analysis: from clouds to exascale

A middleware for parallel processing of large graphs

Scientific data analysis on data-parallel platforms.

Contact Info

Product

Resources

About