Resource Matching in Non-dedicated Multicluster Environments

Abstract

Abstract. We are interested in making use of Multiclusters to execute parallel applications. The present work is developed within the M-CISNE project. M-CISNE is a non-dedicated and heterogeneous Multicluster environment which includes MetaLoRaS, a two-level MetaScheduler that manages the appropriate job allocation to available resources. In this paper, we present a new resource-matching model for MetaLoRaS, which is aimed at mitigating the degraded turnaround time of co-allocated jobs, caused by the contention on shared inter-cluster links. The model is linear programming based and considers the availability of computational resources and the contention of shared inter and intra-cluster links. Its goal is to minimize the average turnaround time of the parallel applications without disturbing the local applications excessively and maximize the prediction accuracy. We also present a parallel job model that takes both computation and communication characterizations into account. By doing this, greater accuracy is obtained than in other models only focused on one of these characteristics. Our preliminary performance results indicate that the linear programming model for on-line resource matching is efficient in speed and accuracy and can be successfully applied to co-allocate jobs across different clusters.

Multiple job co-allocation strategy for heterogeneous multi-cluster systems based on linear programming

“…The maximum of these values determines the final job SP j and SC j . In [3], there is a full explanation of the multicluster model and also the SP j and SC j calculation.…”

“…In [3], we presented a Mixed-Integer Programming model (MIP) in order to find the best allocation for a parallel job, i.e. one that minimizes its execution time.…”

“…The scheduler has access to distributed resources across different clusters to allocate those jobs that cannot be assigned to a single cluster. This allocation strategy, known as co-allocation, can maximize the job throughput by reducing the queue waiting times, and thus its response times [3]. However, mapping jobs across the cluster boundaries can result in rather poor overall performance when co-allocated jobs contend for inter-cluster network bandwidth.…”

“…Another point of view is presented in [4], which characterizes the bandwidth requirements of the co-allocated jobs minimizing the inter-cluster links usage. This work was extended in [3], adding the minimization of the execution time of the parallel jobs, while preventing the saturation of the inter-cluster links.…”