Resource requirement prediction using clone detection technique

Sarkar, Mitali; Mondal, Triparna; Roy, Sarbani; Mukherjee, Nandini

doi:10.1016/j.future.2012.09.010

Cited by 10 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This makes them amenable to history-based resource prediction [35]. We informally validate this by running seasonality detection and building predictors for workflows from internal Microsoft clusters.…”

Section: Reservation Definition Language (Rdl)mentioning

confidence: 96%

Reservation-based Scheduling

Curino

Difallah

Douglas

et al. 2014

Proceedings of the ACM Symposium on Cloud Computing

101

View full text Add to dashboard Cite

The continuous shift towards data-driven approaches to business, and a growing attention to improving return on investments (ROI) for cluster infrastructures is generating new challenges for big-data frameworks. Systems originally designed for big batch jobs now handle an increasingly complex mix of computations. Moreover, they are expected to guarantee stringent SLAs for production jobs and minimize latency for best-effort jobs.In this paper, we introduce reservation-based scheduling, a new approach to this problem. We develop our solution around four key contributions: 1) we propose a reservation definition language (RDL) that allows users to declaratively reserve access to cluster resources, 2) we formalize planning of current and future cluster resources as a Mixed-Integer Linear Programming (MILP) problem, and propose scalable heuristics, 3) we adaptively distribute resources between production jobs and best-effort jobs, and 4) we integrate all of this in a scalable system named Rayon, that builds upon Hadoop / YARN.We evaluate Rayon on a 256-node cluster against workloads derived from Microsoft, Yahoo!, Facebook, and Cloudera's clusters. To enable practical use of Rayon, we opensourced our implementation as part of Apache Hadoop 2.6.

show abstract