Data Prefetching for Scientific Workflow Based on Hadoop

Chen, Gaozhao; Wu, Shaochun; Gu, Rongrong; Xu, Yongquan; Xu, Lingyu; Ge, Yunwen; Song, Cuicui

doi:10.1007/978-3-642-30454-5_6

Cited by 5 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The method only deal with intra-block prefetching for map tasks. A prefetching stratrgy [6] only focuses on rack-off map tasks and prefetches input data from different rack. So it improves the performance lower than our method.…”

Section: Related Workmentioning

confidence: 99%

Scheduling algorithm based on prefetching in MapReduce clusters

Sun

Zhuang

et al. 2016

Applied Soft Computing

View full text Add to dashboard Cite

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

show abstract

Section: Related Workmentioning

confidence: 99%

Scheduling algorithm based on prefetching in MapReduce clusters

Sun

Zhuang

et al. 2016

Applied Soft Computing

View full text Add to dashboard Cite

show abstract

“…It is a framework that allows distributed processing of large data sets across single servers or thousands of machines by using simple programming models. As presented in [8], the Java open source library is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster.…”

Section: B Scientific Languagesmentioning

confidence: 99%

“…Notable characteristics of scientific computing are data integration, data manipulations during calculation, scientific analysis, data migration and the data store on distributed machines according to guidelines and logical relations [8]. We assert that Web Services can be used to unlock heterogeneous scientific systems to extract and integrate environmental data.…”

Section: Motivation and Concept For An Environmental Workflowmentioning

confidence: 99%

Towards a Service-Based Framework for Environmental Data Processing

Madjarov¹,

Kocijan²,

Grancharova³

et al. 2014

IJACSA

View full text Add to dashboard Cite

Abstract-Scientists are confronted with significant data management problems due to the huge volume and high complexity of environmental data. An important aspect of environmental data management is that data, needed for a process, are not always in the adequate format. In this contribution, we analyze environmental data structure, and model this data using a semantic-based method. Using this model, we design and implement a framework based on Web services for transformation between massive environmental text-based data and relational databases. We present a mapping model for environmental data transformation to be used in the scenario devoted to the methodology for development of stochastic models for prediction of environmental parameters by application of Gaussian processes.

show abstract

Effective resource multiplexing for scientific workflows

Ryan

Lee

2015

2015 17th Asia-Pacific Network Operations and Management Symposium (APNOMS)

View full text Add to dashboard Cite

Data Prefetching for Scientific Workflow Based on Hadoop

Cited by 5 publications

References 7 publications

Scheduling algorithm based on prefetching in MapReduce clusters

Scheduling algorithm based on prefetching in MapReduce clusters

Towards a Service-Based Framework for Environmental Data Processing

Effective resource multiplexing for scientific workflows

Contact Info

Product

Resources

About