BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Xue, Ruini; Gao, Shengli; Ao, Lixiang; Guan, Zhongyang

doi:10.1109/ispdc.2015.12

Cited by 15 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Balancing the load of the system should be performed to improve the performance, and the main criteria in improving the performance are execution time and resource utilization. Their research results show that the efficient solution for having an optimized load balancing system is a good slot allocation strategy that allows a balanced workload 16–18 . It is the concern that we follow in this article.…”

Section: Related Workmentioning

confidence: 98%

“…Their research results show that the efficient solution for having an optimized load balancing system is a good slot allocation strategy that allows a balanced workload. [16][17][18] It is the concern that we follow in this article. Our solution is not dependent on the scheduling algorithm; the load portion's accurate calculation is the primary concern.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Optimized load balancing in high‐performance computing for big data analytics

Mirtaheri

Grandinetti

2021

Concurrency and Computation

View full text Add to dashboard Cite

New generation application problems in big data and high-performance computing (HPC) areas claim very diverse operational properties. The convergence requires the dynamic behavior of system components. Load balancing is a critical issue in response to the highly unpredictable, dynamic, and data-oriented behavior of the system. Possible practical constraints such as communication and load transfer delays play an essential role in designing a dynamic load balancer. On the other hand, according to most of the new platforms' distributed nature, the load balancer should be able to perform in a fully distributed manner. In this research, we consider practical issues, including different processing power, storage capability, communication, load transfer delays, and propose two distributed and optimized load balancing methods in HPC for Big Data processing. We model the constraints and present an argument named compensating factor for the optimized load balancer. We try to minimize the task execution time by reducing the nodes' idle time. We evaluate the proposed methods in different scenarios by using Monte Carlo. Evaluations results show that proposed methods decrease idle time significantly while being scalable to network size and applicable in heterogeneous networks with dynamic resources and configuration. K E Y W O R D Sbig data, distributed computing, high-performance computing, load balancing, optimization INTRODUCTIONAt present, a large amount of data is being generated exponentially due to the massive number of sensors, the Internet of Things (IoT), and connected devices. There are various big data sources such as social media, black box data, stock exchange data, power grid data, transport data, and search engine data that need a huge amount of processing power almost in real-time scenarios. These types of data and information on this scale require to be managed by runtime tools. High-performance computing (HPC) systems are actual solutions for vast and complex processing. However, traditional HPC tools are not adequate, and runtime tools are needed for big data processing on HPC platforms. 1Big data refers to the emerging technologies designed to extract value from data having at least three Vs. namely, volume, variety, and velocity.We can say that "big data" is a collection of large amounts of information with increasing capacity, stored a huge volume of data that can be structured, semi-structured, unstructured, and time-stamped. The statistical and regression techniques may be used for the analysis of this amount of data. 2One of the most important and challenging tools in HPC platforms is the load balancer. The load balancer divides the processing load among available computing systems so that processing work is performed in the minimum time, considering the given constraints. In load balancing, the

show abstract

Section: Related Workmentioning

confidence: 98%

Section: Related Workmentioning

confidence: 99%

Optimized load balancing in high‐performance computing for big data analytics

Mirtaheri

Grandinetti

2021

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…However, this work uses flow network, rather than more popular topology network to model data placement. Work in Reference 55 proposes BOLAS, a task scheduling algorithm, which models the scheduling process as a bipartite‐graph matching problem to assign data block to the nearest task. BOLAS solves the model using Kuhn‐Munkres (KM) optimal matching algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…KM algorithm is a combinatorial optimization algorithm that solves the assignment problem in polynomial time which anticipates latest primal‐dual methods 72 . KM algorithm is widely used in weighted bipartite graph model to realize task scheduling, such as References 54,55,73. The KM algorithm 72 is an efficient way to find the maximum weight perfect matching in a weighted bipartite graph and find a good feasible labeling that remains enough edges in graph.…”

Section: A Data Locality Aware Task Schedulermentioning

confidence: 99%

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

Yang

Grundy

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary Scheduling and resource allocation in clouds is used to harness the power of the underlying resource pool. Service providers can meet quality of service (QoS) requirements of tenants specified in Service Level Agreements. Improving resource allocation ensures that all tenants will receive fairer access to system resources, which improves overall utilization and throughput. Real‐time applications and services require critical deadlines in order to guarantee QoS. A growing number of data‐intensive applications drive the optimization of scheduling through utilizing data locality in which the scheduler locates a task and ensures the task's relevant data to be on the same server. Choosing suitable scheduling mechanisms for running applications that support multitenancy has consistently been a major challenge. This work proposes a new adaptive Deadline constrained and Data locality aware Dynamic Scheduling Framework “ 3DSF“ that orchestrates different schedulers based on varied requirements. This framework considers tenants' deadline‐based QoS requirements, cloud system's performance and a method of resource allocation to improve resource utilization, system throughput and reduce jobs' completion time. 3DSF contains: (a) a real‐time, preemptive, deadline constrained job scheduler, (b) an optimized data locality aware scheduler, (c) an improved Dominant Resource Fairness greedy resource allocation approach, and (d) an adaptive suite to integrate above‐mentioned schedulers together.

show abstract

“…Processing data within a requesting node for a dataintensive application represents a key factor to improve the scheduling performance in Hadoop. Many researchers (e.g., [56] [59] [60] [61]) have been working extensively to solve this problem by evaluating the impact of many factors on the data locality. This can help identify the correlation between data locality and those identified factors and hence schedule tasks on the processing nodes as close as possible to their input data.…”

Section: Data Locality-aware Schedulingmentioning

confidence: 99%

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Soualhia

Khomh

Tahar

2017

Journal of Systems and Software

View full text Add to dashboard Cite

Context: Hadoop, Spark, Storm, and Mesos are very well known frameworks in both research and industrial communities that allow expressing and processing distributed computations on massive amounts of data. Multiple scheduling algorithms have been proposed to ensure that short interactive jobs, large batch jobs, and guaranteed-capacity production jobs running on these frameworks can deliver results quickly while maintaining a high throughput. However, only a few works have examined the effectiveness of these algorithms. Objective: The Evidence-based Software Engineering (EBSE) paradigm and its core tool, i.e., the Systematic Literature Review (SLR), have been introduced to the Software Engineering community in 2004 to help researchers systematically and objectively gather and aggregate research evidences about different topics. In this paper, we conduct a SLR of task scheduling algorithms that have been proposed for big data platforms. Method: We analyse the design decisions of different scheduling models proposed in the literature for Hadoop, Spark, Storm, and Mesos over the period between 2005 and 2016. We provide a research taxonomy for succinct classification of these scheduling models. We also compare the algorithms in terms of performance, resources utilization, and failure recovery mechanisms. Results: Our searches identifies 586 studies from journals, conferences and workshops having the highest quality in this field. This SLR reports about different types of scheduling models (dynamic, constrained, and adaptive) and the main motivations behind them (including data locality, workload balancing, resources utilization, and energy efficiency). A discussion of some open issues and future challenges pertaining to improving the current studies is provided.

show abstract

BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Cited by 15 publications

References 26 publications

Optimized load balancing in high‐performance computing for big data analytics

Optimized load balancing in high‐performance computing for big data analytics

An efficient deadline constrained and data locality aware dynamic scheduling framework for multitenancy clouds

Task Scheduling in Big Data Platforms: A Systematic Literature Review

Contact Info

Product

Resources

About