Scalable parallel computing on clouds using Twister4Azure iterative MapReduce

Gunarathne, Thilina; Zhang, Bingjing; Wu, Tak-Lon; Qiu, Judy

doi:10.1016/j.future.2012.05.027

Cited by 48 publications

(31 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will also extend current work to include an allreduce collective that is an alternative approach to Kmeans. The resultant Map-Collective model that captures the full range of traditional MapReduce and MPI features will be evaluated on Azure [22] as well as IaaS/HPC environments. We will integrate Twister with Infiniband RDMA based protocol and compare various communication scenarios.…”

Section: Discussionmentioning

confidence: 99%

“…Spark has "broadcast" and "gather". Our Twister4Azure system [22] supports "allgather" and "allreduce" and in a later paper we will describe the integration of these different collectives into a single system that runs interoperably on HPC clusters (Twister) or PaaS cloud systems (Twister4Azure) changing the implementation to optimize performance for each infrastructure. The same high level collective primitive is used on each platform with different under-the-hood optimizations.…”

Section: Collective Communication and Intermediate Data Handlingmentioning

confidence: 99%

See 1 more Smart Citation

High performance clustering of social images in a map-collective programming model

Zhang

Qiu

2013

Proceedings of the 4th Annual Symposium on Cloud Computing

Self Cite

View full text Add to dashboard Cite

Large-scale iterative computations are common in many important data mining and machine learning algorithms needed in analytics and deep learning. In most of these applications, individual iterations can be specified as MapReduce computations, leading to the Iterative MapReduce programming model for efficient execution of data-intensive iterative computations interoperably between HPC and cloud environments. Further one needs additional communication patterns from those familiar in MapReduce and we base our initial architecture on collectives that integrate capabilities developed by the MPI and MapReduce communities. This leads us to the MapCollective programming model which here we develop based on requirements of a range of applications by extending our existing Iterative MapReduce environment Twister. This paper studies the implications of large scale Social Image clustering where large scale problems study 10-100 million images represented as points in a high dimensional (up to 2048) vector space which need to be divided into up to 1-10 million clusters. This Kmeans application needs 5 stages in each iteration: Broadcast, Map, Shuffle, Reduce and Combine, and this paper focuses on collective communication stages where large data transfers demand performance optimization. By comparing and combining ideas from MapReduce and MPI communities, we show that a topologyaware and pipeline-based broadcasting method gives better performance than other MPI and (Iterative) MapReduce systems.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Collective Communication and Intermediate Data Handlingmentioning

confidence: 99%

High performance clustering of social images in a map-collective programming model

Zhang

Qiu

2013

Proceedings of the 4th Annual Symposium on Cloud Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Microsoft Daytona 24 proposes an iterative Map-Reduce runtime for Windows Azure to support data analytics and machine learning algorithms. Twister [58] is an enhanced MapReduce runtime with an extended programming model for iterative Map-Reduce computations. Hadoop [20] is the most popular open source implementation of Map-Reduce on top of HDFS, as said in the previous section.…”

Section: Parallel Runtime Environmentsmentioning

confidence: 99%

Big Data Management: What to Keep from the Past to Face Future Challenges?

2017

View full text Add to dashboard Cite

The emergence of new hardware architectures, and the continuous production of data open new challenges for data management. It is no longer pertinent to reason with respect to a predefined set of resources (i.e., computing, storage and main memory). Instead, it is necessary to design data processing algorithms and processes considering unlimited resources via the ''pay-as-you-go'' model. According to this model, resources provision must consider the economic cost of the processes versus the use and parallel exploitation of available computing resources. In consequence, new methodologies, algorithms and tools for querying, deploying and programming data management functions have to be provided in scalable and elastic architectures that can cope with the characteristics of Big Data aware systems (intelligent systems, decision making, virtual environments, smart cities, drug personalization). These functions, must respect QoS properties (e.g., security, reliability, fault tolerance, dynamic evolution and adaptability) and behavior properties (e.g., transactional execution) according to application requirements. Mature and novel system architectures propose models and mechanisms for adding these properties to new efficient data management and processing functions delivered as services. This paper gives an overview of the different architectures in which efficient data management functions can be delivered for addressing Big Data processing challenges.

show abstract

“…In [16] an integrated task and data placement algorithm for workflows on clouds, based on graph partitioning is derived, with the goal of minimizing data transfers. The approach to use data locality for efficient task scheduling is also widely applied to MapReduce, where various improvements over default Hadoop scheduling are proposed [27]. Bharathi et al [10] analyze the impact of data staging strategies on workflow execution on clouds.…”

Section: Related Workmentioning

confidence: 99%

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

Bryk¹,

Malawski

Juve³

et al. 2015

J Grid Computing

View full text Add to dashboard Cite

This paper focuses on data-intensive workflows and addresses the problem of scheduling workflow ensembles under cost and deadline constraints in Infrastructure as a Service (IaaS) clouds. Previous research in this area ignores file transfers between workflow tasks, which, as we show, often have a large impact on workflow ensemble execution. In this paper we propose and implement a simulation model for handling file transfers between tasks, featuring the ability to dynamically calculate bandwidth and supporting a configurable number of replicas, thus allowing us to simulate various levels of congestion. The resulting model is capable of representing a wide range of storage systems available on clouds: from inmemory caches (such as memcached), to distributed file systems (such as NFS servers) and cloud storage (such as Amazon S3 or Google Cloud Storage). We observe that file transfers may have a significant impact on ensemble execution; for some applications up to 90 % of the execution time is spent on file transfers. Next, we propose and evaluate a novel scheduling algorithm that minimizes the number of transfers by taking advantage of data caching and file locality. We find that for data-intensive applications it performs better than other scheduling algorithms. Additionally, we modify the original scheduling algorithms to effectively operate in environments where file transfers take non-zero time.

show abstract

Scalable parallel computing on clouds using Twister4Azure iterative MapReduce

Cited by 48 publications

References 17 publications

High performance clustering of social images in a map-collective programming model

High performance clustering of social images in a map-collective programming model

Big Data Management: What to Keep from the Past to Face Future Challenges?

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

Contact Info

Product

Resources

About