Portable Parallel Programming on Cloud and HPC: Scientific Applications of Twister4Azure

Gunarathne, Thilina; Zhang, Bingjing; Wu, Tak-Lon; Qiu, Judy

doi:10.1109/ucc.2011.23

Cited by 14 publications

(11 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples of applications that can be implemented using iterative MapReduce include PageRank, Multi-Dimensional Scaling [3,17], K-means Clustering, Descendent query [7], LDA, and Collaborative Filtering with ALS-WR.…”

Section: Iterative Mapreduce and Twister4azurementioning

confidence: 99%

“…However, there exist many possible optimizations and programming model improvements to improve the performance and usability of the iterative MapReduce programs. Such optimization opportunities are highlighted by the development of many iterative MapReduce frameworks such as Twister [6], HaLoop [7], Twister4Azure [3], Daytona [18] and spark [19]. Optimizations exploited by these frameworks include caching of loopinvariant data, cache aware scheduling of tasks, iterative aware programming models, direct memory streaming of intermediate data, iteration-aware fault tolerance, caching of intermediate data (HaLoop reducer input cache), dynamic modifications to cached data (e.g.…”

Section: Iterative Mapreduce and Twister4azurementioning

confidence: 99%

“…Twister4Azure optimizes the iterative MapReduce computations by multi-level caching of loop invariant data, performing cache aware scheduling, optimizing intermediate data transfers, optimizing data broadcasts and many other optimizations described in Gunarathne et al [3].…”

Section: Iterative Mapreduce and Twister4azurementioning

confidence: 99%

See 2 more Smart Citations

Towards a Collective Layer in the Big Data Stack

Gunarathne

Qiu

Gannon

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

Section: Iterative Mapreduce and Twister4azurementioning

confidence: 99%

Section: Iterative Mapreduce and Twister4azurementioning

confidence: 99%

See 1 more Smart Citation

Towards a Collective Layer in the Big Data Stack

Gunarathne

Qiu

Gannon

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

“…Iterative MapReduce is a programming model that can have the performance of MPI and the fault tolerance and dynamic flexibility of the original MapReduce. Open source Java Twister [4,5] and Twister4Azure [6,7] have been released as an Iterative MapReduce framework. …”

Section: Mapping Applications To Cloudsmentioning

confidence: 99%

Data intensive applications on clouds

Fox

2011

Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds

View full text Add to dashboard Cite

The cyberinfrastructure supporting science will include large-scale simulation systems headed to exascale combined with cloud like systems supporting data intensive and high throughput computing, pleasingly parallel jobs and the long tail of science. Clouds offer economies of scale, elasticity supporting real time and interactive use and powerful new programming models such as MapReduce. We stress that iterative extensions of MapReduce will be necessary to get good performance on for several data mining (analytics) applications. We give several illustrations mainly from bioinformatics. We suggest that the data deluge implies a corresponding increase in the computational resources needed to support analysis and this suggests new architectures for large scale data repositories.

show abstract

“…Open source Java Twister [2][3] and Twister4Azure [4][5] have been released as Iterative MapReduce framework based on our initial research on data-intensive programming models and their runtime. Twister interpolates between MPI and MapReduce and, suitably configured, can mimic their characteristics, and, more interestingly, can be positioned as a programming model that has the performance of MPI and the fault tolerance and dynamic flexibility of the original MapReduce.…”

Section: Architecture Overviewmentioning

confidence: 99%

Generalizing mapreduce as a unified cloud and HPC runtime

Qiu

2011

Proceedings of the 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities

View full text Add to dashboard Cite

Computational simulation and analysis were one of the keys to the future in data-intensive science as a "fourth paradigm" of scientific discovery but facing a major challenge as handling the incredible increases in dataset sizes. This requires attractive powerful programming models that address issues of portability with scaling performance and fault tolerance. Further, one must meet these challenges for both computation and storage. We build on the success of our research on Iterative MapReduce with successful prototypes Twister (on HPC) and Twister4Azure (on clouds). We have designed a novel Map Collective runtime which generalizes previous work in both HPC and MapReduce communities, which we hypothesize can be used as the runtime for data analysis (mining) interoperably between clouds and clusters.

show abstract

Portable Parallel Programming on Cloud and HPC: Scientific Applications of Twister4Azure

Cited by 14 publications

References 14 publications

Towards a Collective Layer in the Big Data Stack

Towards a Collective Layer in the Big Data Stack

Data intensive applications on clouds

Generalizing mapreduce as a unified cloud and HPC runtime

Contact Info

Product

Resources

About