Flexible Scheduling of Distributed Analytic Applications

Abstract

This work addresses the problem of scheduling user-defined analytic applications, which we define as high-level compositions of frameworks, their components, and the logic necessary to carry out work. The key idea in our application definition, is to distinguish classes of components, including rigid and elastic types: the first being required for an application to make progress, the latter contributing to reduced execution times. We show that the problem of scheduling such applications poses new challenges, which existing approaches address inefficiently.Thus, we present the design and evaluation of a novel, flexible heuristic to schedule analytic applications, that aims at high system responsiveness, by allocating resources efficiently. Our algorithm is evaluated using trace-driven simulations, with large-scale real system traces: our flexible scheduler outperforms a baseline approach across a variety of metrics, including application turnaround times, and resource allocation efficiency.We also present the design and evaluation of a full-fledged system, which we have called Zoe, that incorporates the ideas presented in this paper, and report concrete improvements in terms of efficiency and performance, with respect to prior generations of our system.

A Flexible Heuristic to Schedule Distributed Analytic Applications in Compute Clusters

“…This article extends the work presented in [27] in several respects: (i) we implement a malleable scheduler, and compared the results with our solution; (ii) we study different definitions of job size and discuss their impact on the perfomance indexes; (iii) we implement and evaluate other scheduling policies that fall in the category of SMART policies, such as SRPT (Shortest-Remaining-Processing-Time) and HRRN (Higher-Response-Ratio Next).…”

“…We have tested our scheduler with all the policies mentioned above (SJF, SRPT, HRRN), in all the different scenarios: comparison with the rigid, as well as malleable schedule, when no interactive applications are present; comparison with the different definitions of size; comparison when preemption is enabled, comparison with different workloads. We do not report here all the set of results since they yield the same information of the results presented in the previous sections -the interested reader can find them in our Technical Report [45]. In summary, our flexible scheduler is able to reduce the turnaround time, while improving resources allocation, in all the different policies.…”

“…Next, since that definition is one dimensional, we try to move 7. More information and experiments can be found in [45] to a two-dimensional definition by adding also the number of components that the application is using. Finally, since every component might request different resources, we add this information as well, and turn the definition from a two to a three dimension.…”

A Data-Driven Approach to Dynamically Adjust Resource Allocation for Compute Clusters

“…The backend module is an instance of a cluster management system, such as Docker [18] or Kubernetes [26]. Additionally, we assume the presence of an application scheduler such as [42], which reads the compute cluster state from a dedicated database component. Finally, the monitoring component populates the cluster state database with measurements taken from the backend.…”

“…We use such information to dynamically adapt the safe-guard buffer that should prevent application failures. In addition, the frameworks, on which the applications are based, are composed by several elements that are characterized by either a core or elastic nature [42]. Core components are compulsory for a framework to produce useful work (e.g, Apache Spark requires a controller, a master, and one worker); elastic components, instead, optionally contribute to a job, e.g.…”