Multihybrid job scheduling for fault-tolerant distributed computing in policy-constrained resource networks

Moon, Yong-Hyuk; Youn, Chan‐Hyun

doi:10.1016/j.comnet.2015.02.030

Cited by 17 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, job-level scheduling in the hybrid cloud environment has produced noteworthy research results. The research in [4] proposed a fault-tolerant scheduling strategy in Hadoop, which optimizes completion times, maximizes resource utilization, and minimizes the task failure rate. In [5], a new Hadoop scheduling strategy, COSHH, was presented, and it considers heterogeneity at both the application and cluster levels.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Job-Scheduling Algorithm based on Queuing Theory in a Hybrid Cloud Environment

Liu¹

2019

IJPE

View full text Add to dashboard Cite

To resolve the problem of unreasonable resource allocations caused by the continuous arrival of different types of jobs in a hybrid cloud environment, an adaptive job-scheduling algorithm based on queuing theory is proposed. This paper analyses job load types, and the jobs are classified according to the logistic regression method. A resource utility is used to classify the nodes in a private cloud cluster by considering the heterogeneity of the private cloud resources. Based on the job classification and the resource classification, a queuing model is established, and an adaptive genetic algorithm is used to manage the job queue's arrival rate that becomes the basis of the resource allocation. The proposed algorithm is compared with some existed similar algorithms to verify its performance in terms of job response times and throughput.

show abstract

Section: Introductionmentioning

confidence: 99%

Adaptive Job-Scheduling Algorithm based on Queuing Theory in a Hybrid Cloud Environment

Liu¹

2019

IJPE

View full text Add to dashboard Cite

show abstract

“…According to in Moon and Youn (2015) 70–75% resources have failure rates of around 20 and 40% in workload archives such as DEUB, UCB and SDSC (Kondo et al 2010). Furthermore, their application level traces reveal that most of their resources have more failure probabilities which further cause issues related to performance of scheduling and unavailability of resources (Kondo et al 2010; Li et al 2006).…”

Section: Introductionmentioning

confidence: 99%

Fault tolerance in computational grids: perspectives, challenges, and issues

Haider

Nazir

2016

SpringerPlus

View full text Add to dashboard Cite

Computational grids are established with the intention of providing shared access to hardware and software based resources with special reference to increased computational capabilities. Fault tolerance is one of the most important issues faced by the computational grids. The main contribution of this survey is the creation of an extended classification of problems that incur in the computational grid environments. The proposed classification will help researchers, developers, and maintainers of grids to understand the types of issues to be anticipated. Moreover, different types of problems, such as omission, interaction, and timing related have been identified that need to be handled on various layers of the computational grid. In this survey, an analysis and examination is also performed pertaining to the fault tolerance and fault detection mechanisms. Our conclusion is that a dependable and reliable grid can only be established when more emphasis is on fault identification. Moreover, our survey reveals that adaptive and intelligent fault identification, and tolerance techniques can improve the dependability of grid working environments.

show abstract

“…(12) The skewed partition detection algorithm (SPD) consists of three steps: first, it evaluates Reducer's computational capability based on the Reducer task log information successfully executed on each Reducer; after that, it calculates the load distribution threshold for each Reducer based on the objective of balancing the skewed Reduce input load; finally, it detects whether there is any skewed partition according to the load allocation threshold of each Reducer and the sizes of virtual partitions corresponding to all Map task output data [14][15].…”

Section: Heterogeneity-aware Loads Balancingmentioning

confidence: 99%

Job Performance Optimization Method Based on Data Balance in the Wireless Sensor Networks

Zhao

2017

Int. J. Onl. Eng.

View full text Add to dashboard Cite

Abstract-In the wireless sensor network, the representative MapReduce computing model based on data center has been widely used in large-scale data processing. In the data transmission phase, the wireless sensor network system uses the hash method to distribute data for each Reduce task based on the number of Reduce tasks. This data partitioning method based on the hash function results in non-uniform distribution of the output data in the data transmission phase and further leads to skewing of the input data in the Reduce task. Data skew will result in load imbalance in the Reduce phase and causes the system performance to degrade. In order to eliminate the data skew problem in the Reduce phase, this paper presents a load balancing method, which consists of two parts: the virtual partitioning method based on the consistent hashing and the heterogeneity-aware loads balancing (HLB) algorithm. The experimental results show that the proposed method can eliminate the data skew in the Reduce phase and distribute the load equitably for each Reduce task. In addition, the method produces less system overhead.

show abstract

Multihybrid job scheduling for fault-tolerant distributed computing in policy-constrained resource networks

Cited by 17 publications

References 19 publications

Adaptive Job-Scheduling Algorithm based on Queuing Theory in a Hybrid Cloud Environment

Adaptive Job-Scheduling Algorithm based on Queuing Theory in a Hybrid Cloud Environment

Fault tolerance in computational grids: perspectives, challenges, and issues

Job Performance Optimization Method Based on Data Balance in the Wireless Sensor Networks

Contact Info

Product

Resources

About