Medea

Abstract

The rise in popularity of machine learning, streaming, and latencysensitive online applications in shared production clusters has raised new challenges for cluster schedulers. To optimize their performance and resilience, these applications require precise control of their placements, by means of complex constraints, e.g., to collocate or separate their long-running containers across groups of nodes. In the presence of these applications, the cluster scheduler must attain global optimization objectives, such as maximizing the number of deployed applications or minimizing the violated constraints and the resource fragmentation, but without affecting the scheduling latency of short-running containers.We present Medea, a new cluster scheduler designed for the placement of long-and short-running containers. Medea introduces powerful placement constraints with formal semantics to capture interactions among containers within and across applications. It follows a novel two-scheduler design: (i) for long-running containers, it applies an optimization-based approach that accounts for constraints and global objectives; (ii) for short-running containers, it uses a traditional task-based scheduler for low placement latency. Evaluated on a 400-node cluster, our implementation of Medea on Apache Hadoop YARN achieves placement of long-running applications with significant performance and resilience benefits compared to state-of-the-art schedulers. CCS CONCEPTS• Computer systems organization → Distributed architectures; • Software and its engineering → Scheduling; Cloud computing; • Theory of computation → Linear programming;

Optimizing Quality-Aware Big Data Applications in the Cloud

“…A similar approach is proposed in [47], where the authors present a solution to manage clusters shared among Hadoop applications and more traditional Web systems. Another recent contribution addresses the scheduling of long-running applications in shared clusters [52]. The authors couple a task-based scheduler, to cope with short jobs, and an ILP formulation that takes care of scheduling long-running applications while minimizing costs and violations of placement constraints, thus achieving up to 32% shorter median execution time in comparison to a baseline constraint-aware scheduler.…”

QoS for best‐effort batch jobs in container‐based cloud

“…Paragon 9 and Quasar 10 predict and detect the resource interference between jobs through workload profiling and application classification, and select the optimal node accordingly. In addition, there were studies that supported granular partitioning to analyze and eliminate the cases of interference between latency‐sensitive jobs and best‐effort batch jobs when deploying the application to the node 11‐16 . These studies focused on resource guarantee and efficient idle resource consumption.…”

“…In addition, there were studies that supported granular partitioning to analyze and eliminate the cases of interference between latency-sensitive jobs and best-effort batch jobs when deploying the application to the node. [11][12][13][14][15][16] These studies focused on resource guarantee and efficient idle resource consumption. On the other hand, detailed control of the resources used by best-effort batch jobs with a low priority was not studied thoroughly.…”

“…Thus, it is highly required to provide more delicate QoS control for best‐effort batch jobs. However, since the performance of high‐priority jobs is deservingly more important than that of low‐priority jobs, existing studies on QoS in the cloud have focused on resource guarantee and isolation of high‐priority jobs 6‐16 . In this paper, we aim at providing QoS for best‐effort batch jobs, while preventing side effects that may occur to high‐priority jobs.…”

“…However, since the performance of high-priority jobs is deservingly more important than that of low-priority jobs, existing studies on QoS in the cloud have focused on resource guarantee and isolation of high-priority jobs. [6][7][8][9][10][11][12][13][14][15][16] In this paper, we aim at providing QoS for best-effort batch jobs, while preventing side effects that may occur to high-priority jobs. We particularly target the container-based cloud.…”