An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing

Niu, Zhaojie; Tang, Shanjiang; He, Bingsheng

doi:10.1109/tsc.2016.2635133

Cited by 10 publications

(7 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adaptive approaches achieve performance increases through the adaptation of the scheduling policy or architecture at runtime, providing the benefit of an alternate scheduling approach autonomously without the limitations of hybrid approaches requiring two separate approaches simultaneously. However, current adaptive approaches to scheduling are limited by their need for user provided parameters to inform scheduling decisions [12], or the need for user-generated models [13]. The need for domain-specific knowledge and user intervention for informing adaptation limits the approach as scheduling may only be improved and maintained across all encountered workloads with sufficient data and correct models for all.…”

Section: Adaptive and Learned Schedulersmentioning

confidence: 99%

“…Workload-specific schedulers, such as that of Ray [8], focus on improving end-to-end performance for a specific workload type, for which a general-purpose scheduler would perform poorly. Hybrid (e.g., [9]- [11]) and adaptive [2], [12], [13] approaches attempt to gain the best of both by simultaneously employing multiple different scheduling architectures or policies. However, to date they use hand-crafted rules or expect the user to submit detailed specifications with their data processing job to enable selection of the ideal scheduler -or alternatively use offline training of a policy which is then fixed at runtime.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Design Space of Emergent Scheduling for Distributed Execution Frameworks

Dean

Porter

2021

2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)

View full text Add to dashboard Cite

Distributed Execution Frameworks (DEFs) such as Apache Spark have become ubiquitous as a solution for the execution of user-defined jobs to process terabytes of data across hundreds of nodes. One of the key costs of DEFs is scheduling of which parts of each job are placed on each host; better scheduling decisions provide lower overall execution time for each job, more efficient resource usage, and reduced energy consumption. Existing DEFs use a static approach to scheduling, either with a single generalised scheduler which aims to be a good fit for most workloads, or with a special-purpose scheduler which is tuned to optimise for a particular kind of workload. In both cases the scheduling implementation is fixed at design-time such that the DEF is unable to adjust to the actual characteristics of workloads that arrive at deployment time. In this paper we introduce an emergent scheduler for Distributed Execution Frameworks. This scheduler can be composed and re-composed at runtime from a set of different building blocks, allowing the system to dynamically provide the benefits of differing scheduling policies over time depending on the actual properties of incoming workloads -with improved performance and resource usage. In this paper we present the overall design of our emergent scheduler, we discuss the theoretical design space of different scheduling approaches, and we examine a specific research question to determine the correlation between workload properties and scheduling performance for different scheduler implementations. Our results are based on a real implementation of our emergent DEF running across multiple hosts in a real datacentre, and our implementation is made available as open-source software.

show abstract

Section: Adaptive and Learned Schedulersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The Design Space of Emergent Scheduling for Distributed Execution Frameworks

Dean

Porter

2021

2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)

View full text Add to dashboard Cite

show abstract

“…Niu et al [25] propose to formally measure the fairness loss F of a scheduler. They consider J (J ∈ {1,..,k}) is the…”

Section: Problem Formulation Examplementioning

confidence: 99%

“…The completion times of the application of a user i under a specific scheduler and a fair scheduler are t * i and t i respectively. Niu et al [25] calculate first the reduction of the completion time of the applications of a user i ; s i as follows:…”

Section: Accepted Manuscriptmentioning

confidence: 99%

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Soualhia

Khomh

Tahar

2017

Journal of Systems and Software

View full text Add to dashboard Cite

Context: Hadoop, Spark, Storm, and Mesos are very well known frameworks in both research and industrial communities that allow expressing and processing distributed computations on massive amounts of data. Multiple scheduling algorithms have been proposed to ensure that short interactive jobs, large batch jobs, and guaranteed-capacity production jobs running on these frameworks can deliver results quickly while maintaining a high throughput. However, only a few works have examined the effectiveness of these algorithms. Objective: The Evidence-based Software Engineering (EBSE) paradigm and its core tool, i.e., the Systematic Literature Review (SLR), have been introduced to the Software Engineering community in 2004 to help researchers systematically and objectively gather and aggregate research evidences about different topics. In this paper, we conduct a SLR of task scheduling algorithms that have been proposed for big data platforms. Method: We analyse the design decisions of different scheduling models proposed in the literature for Hadoop, Spark, Storm, and Mesos over the period between 2005 and 2016. We provide a research taxonomy for succinct classification of these scheduling models. We also compare the algorithms in terms of performance, resources utilization, and failure recovery mechanisms. Results: Our searches identifies 586 studies from journals, conferences and workshops having the highest quality in this field. This SLR reports about different types of scheduling models (dynamic, constrained, and adaptive) and the main motivations behind them (including data locality, workload balancing, resources utilization, and energy efficiency). A discussion of some open issues and future challenges pertaining to improving the current studies is provided.

show abstract

“…However, they do not consider the tradeoff between the resource efficiency, job latency, fairness and energy consumption. Therefore, new schedulers [16,17,18,19] are proposed to address the tradeoff between these discordant factors.…”

Section: Hadoop Yarnmentioning

confidence: 99%

A study of big data computing platforms : performance, fairness and energy consumption

Niu¹

Self Cite

View full text Add to dashboard Cite

Today, population aging is a trend spreading across the world. It is estimated that the number of people aged 60 years or over, is increasing rapidly. Although the elderly has wisdom and wealth gathered from their life experience, they often require long-term assistance from others. Ambient assisted living (AAL) systems open up a new opportunity to address the needs of aged by utilizing information and communication techniques. Multiple sensors and actuators coexist in ambient assisted living systems and are deployed everywhere surrounding elderly. A huge volume of sensed data is collected continuously from elderly, storage of these massive raw data and efficient processing on them to infer knowledge from the data and also guide the actuators to meet the needs of elderly is a problem we face. In the thesis, we mainly focus on the efficient large-scale data processing, we investigate whether and how we can build an effective large-scale data processing system to effectively excavate knowledge, wisdom and skills from elderly people to impact the development of the entire society.A lot of large-scale big data analytics systems have thereby emerged to process the massive data effectively. Efficient job scheduling and resource management for these data analytic frameworks are nontrivial. Modern job schedulers and resource coordinators in data processing frameworks often need to consider multiple objectives simultaneously due to various system operators requirements on data analytics. Currently, resource efficiency (throughput), job latency (per-job performance), fairness (isolation guarantee) and energy consumption are important concerns for the job scheduler in modern large-scale multiple-tenant environments. Resource efficiency is de facto the very important factor for the big data analytic framework. Job latency reflects the waiting time of the application. Fairness is a key building block of any multiple-tenant computing system that allows resource sharing effectively. Energy usage of the data center has reached 3% of the global electricity consumption while generating 200 million metric tons of CO2 in 2014. In order to reduce carbon emission and financial burden on the electricity, a lot of data centers have been re-designed and powered with multiple energy sources, including renewable (green) energy from non-polluting sources and brown energy from traditional polluting sources.Improving the resource efficiency and reducing the per-job latency are the common sense on those large-scale data processing frameworks and fruitful studies are proposed in these directions. In i I would like to thank my friends and colleagues in IGS-LILY and PDSL. I will always remember all the great memories we created together in the part four years.I gratefully acknowledge the scholarship provided by LILY, IGS(Interdisciplinary Graduate School) and Nanyang Technological University for my doctorial program.Last but not least, I want to thank my family and friends for their unwavering support throughput my PhD.

show abstract

An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing

Cited by 10 publications

References 39 publications

The Design Space of Emergent Scheduling for Distributed Execution Frameworks

The Design Space of Emergent Scheduling for Distributed Execution Frameworks

Task Scheduling in Big Data Platforms: A Systematic Literature Review

A study of big data computing platforms : performance, fairness and energy consumption

Contact Info

Product

Resources

About