LiPS: A Cost-Efficient Data and Task Co-Scheduler for MapReduce

Ehsan, Moussa; Sion, Radu

doi:10.1109/ipdpsw.2013.175

Cited by 4 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Big-data schedulers: Different schedulers have been proposed to optimize resource consumption or delay, 33,34 or to enhance data locality. 35 We work with existing schedulers, optimizing inside our framework to mitigate skew.…”

Section: Related Workmentioning

confidence: 99%

Plumb: Efficient stream processing of multi‐user pipelines

Qadeer

Heidemann

2020

Softw Pract Exp

View full text Add to dashboard Cite

Operational services run 24×7 and require analytics pipelines to evaluate performance. In mature services such as domain name system (DNS), these pipelines often grow to many stages developed by multiple, loosely coupled teams. Such pipelines pose two problems: first, computation and data storage may be duplicated across components developed by different groups, wasting resources. Second, processing can be skewed, with structural skew occurring when different pipeline stages need different amounts of resources, and computational skew occurring when a block of input data requires increased resources. Duplication and structural skew both decrease efficiency, increasing cost, latency, or both. Computational skew can cause pipeline failure or deadlock when resource consumption balloons; we have seen cases where pessimal traffic increases CPU requirements 6‐fold. Detecting duplication is challenging when components from multiple teams evolve independently and require fault isolation. Skew management is hard due to dynamic workloads coupled with the conflicting goals of both minimizing latency and maximizing utilization. We propose Plumb, a framework to abstract stream processing as large‐block streaming (LBS) for a multi‐stage, multi‐user workflow. Plumb users express analytics as a DAG of processing modules, allowing Plumb to integrate and optimize workflows from multiple users. Many real‐world applications map to the LBS abstraction. Plumb detects and eliminates duplicate computation and storage, and it detects and addresses both structural and computational skew by tracking computation across the pipeline. We exercise Plumb using the analytics pipeline for B‐Root DNS. We compare Plumb to a hand‐tuned system, cutting latency to one‐third the original, and requiring 39% fewer container hours, while supporting more flexible, multi‐user analytics and providing greater robustness to DDoS‐driven demands.

show abstract

Section: Related Workmentioning

confidence: 99%

Plumb: Efficient stream processing of multi‐user pipelines

Qadeer

Heidemann

2020

Softw Pract Exp

View full text Add to dashboard Cite

show abstract

“…LiPS system is introduced in [21] and promised a cost-efficient data and task co-scheduler for MapReduce in a cloud environment using linear programming technique.…”

Section: Background and Related Workmentioning

confidence: 99%

Evaluating map reduce tasks scheduling algorithms over cloud computing infrastructure

Althebyan

Jararweh

Yaseen

et al. 2015

Concurrency and Computation

View full text Add to dashboard Cite

Efficiently scheduling MapReduce tasks is considered as one of the major challenges that face MapReduce frameworks. Many algorithms were introduced to tackle this issue. Most of these algorithms are focusing on the data locality property for tasks scheduling. The data locality may cause less physical resources utilization in non-virtualized clusters and more power consumption. Virtualized clusters provide a viable solution to support both data locality and better cluster resources utilization. In this paper, we evaluate the major MapReduce scheduling algorithms such as FIFO, Matchmaking, Delay, and multithreading locality (MTL) on virtualized infrastructure. Two major factors are used to test the evaluated algorithms: the simulation time and the energy consumption. The evaluated schedulers are compared, and the results show the superiority and the preference of the MTL scheduler over the other existing schedulers. Also, we present a comparison study between virtualized and non-virtualized clusters for MapReduce tasks scheduling. EVALUATING MAP REDUCE TASKS SCHEDULING ALGORITHMS 5687 as 'datasets whose size are beyond the ability of typical database software tools to capture, store, manage and analyze.' New technologies are needed to be able to extract values from those datasets; such processed data might be used in other fields such as artificial intelligence, data mining, health care, and social networks. International Business Machine researchers [4] characterized big data with the 3Vs: variety, volume, and velocity. Variety is used to refer to the multiple types/formats in which big data is generated such as digits, texts, audios, videos, and log files.The second characteristic is the huge volume of big data which can reach hundreds or thousands of terabytes. The third characteristic is the velocity where processing and analyzing data must be performed in a fast manner to extract value of data within an appropriate time. These characteristics drive for developing new methodologies to deal with such huge amounts of data. So, comes to existence the term 'big data management'.Big data operations are widely used in many technologies, for example, cloud computing, distributed systems, data warehouse, Hadoop, and MapReduce. MapReduce is one of these technologies that are utilized to handle such big data. It is a software framework introduced by Google for processing large amounts of data in a parallel manner [5]. In fact, it provides a set of features such as user-defined functions, automatic parallelization and distribution, fault tolerance, and high availability by data replicating.MapReduce works in two phases: the map phase and the reduce phase. In the map phase, a dedicated node called the master node takes the input, divides it into smaller shared data splits, and assigns them to worker nodes. The worker nodes may perform the same splitting operation, leading to a hierarchal tree structure. The worker node processes the assigned splits and sends the results back to the master node. The reduce phase then begins...

show abstract

Cooperative Job Scheduling and Data Allocation for Busy Data-Intensive Parallel Computing Clusters

Liu

Shen

Wang

2019

Proceedings of the 48th International Conference on Parallel Processing

View full text Add to dashboard Cite

In data-intensive parallel computing clusters, it is important to provide deadline-guaranteed service to jobs while minimizing resource usage (e.g., network bandwidth and energy). Under the current computing framework (that first allocates data and then schedules jobs), in a busy cluster with many jobs, it is difficult to achieve these objectives simultaneously. We model the problem to simultaneously achieve the objectives using integer programming, and propose a heuristic Cooperative job Scheduling and data Allocation method (CSA). CSA novelly reverses the order of data allocation and job scheduling in the current computing framework, i.e., changing data-first-job-second to job-first-data-second. It enables CSA to proactively consolidate tasks with more common requested data to the same server when conducting deadline-aware scheduling, and also consolidate the tasks to as few servers as possible to maximize energy savings. This facilitates the subsequent data allocation step to allocate a data block to the server that hosts most of this data's requester tasks, thus maximally enhancing data locality and reduce bandwidth consumption. CSA also has a recursive schedule refinement process to adjust the job and data allocation schedules to improve system performance regarding the three objectives and achieve the tradeoff between data locality and energy savings with specified weights. We implemented CSA and a number of previous job schedulers on Apache Hadoop on a real supercomputing cluster. Trace-driven experiments in the simulation and the real cluster show that CSA outperforms other schedulers in supplying deadline-guarantee and resource-efficient services.

show abstract

LiPS: A Cost-Efficient Data and Task Co-Scheduler for MapReduce

Cited by 4 publications

References 3 publications

Plumb: Efficient stream processing of multi‐user pipelines

Plumb: Efficient stream processing of multi‐user pipelines

Evaluating map reduce tasks scheduling algorithms over cloud computing infrastructure

Cooperative Job Scheduling and Data Allocation for Busy Data-Intensive Parallel Computing Clusters

Contact Info

Product

Resources

About