A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments

Lee, Chia‐Wei; Hsieh, Kuang Yu; Hsieh, Sun Yuan; Hsiao, Hung Chang

doi:10.1016/j.bdr.2014.07.002

Cited by 62 publications

(34 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While it may not be possible to eliminate data centers from the computing ecosystem, innovative and novel system architectures that can geographically distribute data center computing are required for sustainability. Useful contributions in this space can be achieved by developing algorithms that rely on geographically distributed data coordination, resource provisioning and carbon footprint-aware and energy-aware provisioning in data centers [124,125,126,127]. These will in turn minimise energy consumption of the data center and maximise the use of green energy while meeting an application's QoS expectations.…”

Section: Building Sustainable Infrastructure For the Futurementioning

confidence: 99%

Next generation cloud computing: New trends and research directions

Varghese

Buyya

2018

Future Generation Computer Systems

621

270

View full text Add to dashboard Cite

The landscape of cloud computing has significantly changed over the last decade. Not only have more providers and service offerings crowded the space, but also cloud infrastructure that was traditionally limited to single provider data centers is now evolving. In this paper, we firstly discuss the changing cloud infrastructure and consider the use of infrastructure from multiple providers and the benefit of decentralising computing away from data centers. These trends have resulted in the need for a variety of new computing architectures that will be offered by future cloud infrastructure. These architectures are anticipated to impact areas, such as connecting people and devices, data-intensive computing, the service space and self-learning systems. Finally, we lay out a roadmap of challenges that will need to be addressed for realising the potential of next generation cloud systems.

show abstract

Section: Building Sustainable Infrastructure For the Futurementioning

confidence: 99%

Next generation cloud computing: New trends and research directions

Varghese

Buyya

2018

Future Generation Computer Systems

621

270

View full text Add to dashboard Cite

show abstract

“…Similar work has been done in the work of Lee et al to place data blocks based on the computing capacity of a virtual node in a heterogeneous environment by Lee et al to minimize the data transfer time and improve performance. They observed that for different job types, the computing capacity of virtual nodes is not the same.…”

Section: Literature Surveymentioning

confidence: 78%

“…Xie et al and Anjos et al explore the possibility of placing data blocks to minimize job latency. Data blocks are placed based on the computing ratio in other works, to minimize makespan, whereas Chen et al place data blocks to minimize network transfer time. Anjos et al considers the capacity of nodes to minimize the latency of a job.…”

Section: Literature Surveymentioning

confidence: 99%

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

Jeyaraj

Ananthanarayana

Paul

2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Big data is largely influencing business entities and research sectors to be more data‐driven. Hadoop MapReduce is one of the cost‐effective ways to process large scale datasets and offered as a service over the Internet. Even though cloud service providers promise an infinite amount of resources available on‐demand, it is inevitable that some of the hired virtual resources for MapReduce are left unutilized and makespan is limited due to various heterogeneities that exist while offering MapReduce as a service. As MapReduce v2 allows users to define the size of containers for the map and reduce tasks, jobs in a batch become heterogeneous and behave differently. Also, the different capacity of virtual machines in the MapReduce virtual cluster accommodate a varying number of map/reduce tasks. These factors highly affect resource utilization in the virtual cluster and the makespan for a batch of MapReduce jobs. Default MapReduce job schedulers do not consider these heterogeneities that exist in a cloud environment. Moreover, virtual machines in MapReduce virtual cluster process an equal number of blocks regardless of their capacity, which affects the makespan. Therefore, we devised a heuristic‐based MapReduce job scheduler that exploits virtual machine and MapReduce workload level heterogeneities to improve resource utilization and makespan. We proposed two methods to achieve this: (i) roulette wheel scheme based data block placement in heterogeneous virtual machines, and (ii) a constrained 2‐dimensional bin packing to place heterogeneous map/reduce tasks. We compared heuristic‐based MapReduce job scheduler against the classical fair scheduler in MapReduce v2. Experimental results showed that our proposed scheduler improved makespan and resource utilization by 45.6% and 47.9% over classical fair scheduler.

show abstract

“…In the future, we want to address cases where CAPre offers limited improvement by collecting more information during application execution, while studying the overhead that such a hybrid approach might introduce. We also plan to use the predictions made by the developed static code analysis to apply other performance improvement techniques in conjunction with prefetching, such as smart cache replacement policies [39,40,41] and dynamic data placement [42,43].…”

Section: Discussionmentioning

confidence: 99%

CAPre: Code-Analysis based Prefetching for Persistent Object Stores

Touma

Queralt

Cortés

2020

Future Generation Computer Systems

View full text Add to dashboard Cite

Data prefetching aims to improve access times to data storage systems by predicting data records that are likely to be accessed by subsequent requests and retrieving them into a memory cache before they are needed. In the case of Persistent Object Stores, previous approaches to prefetching have been based on predictions made through analysis of the store's schema, which generates rigid predictions, or monitoring access patterns to the store while applications are executed, which introduces memory and/or computation overhead.In this paper, we present CAPre, a novel prefetching system for Persistent Object Stores based on static code analysis of object-oriented applications. CAPre generates the predictions at compile-time and does not introduce any overhead to the application execution. Moreover, CAPre is able to predict large amounts of objects that will be accessed in the near future, thus enabling the object store to perform parallel prefetching if the objects are distributed, in a much more aggressive way than in schema-based prediction algorithms. We integrate CAPre into a distributed Persistent Object Store and run a series of experiments that show that it can reduce the execution time of applications from 9% to over 50%, depending on the nature of the application and its persistent data model.

show abstract

A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments

Cited by 62 publications

References 9 publications

Next generation cloud computing: New trends and research directions

Next generation cloud computing: New trends and research directions

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

CAPre: Code-Analysis based Prefetching for Persistent Object Stores

Contact Info

Product

Resources

About