Mixed Data-Parallel Scheduling for Distributed Continuous Integration

Abstract

In this paper, we consider the problem of scheduling a special kind of mixed data-parallel applications arising in the context of continuous integration. Continuous integration (CI) is a software engineering technique, which consists in rebuilding and testing interdependent software components as soon as developers modify them. The CI tool is able to provide quick feedback to the developers, which allows them to fix the bug soon after it has been introduced. The CI process can be described as a DAG where nodes represent package build tasks, and edges represent dependencies among these packages; build tasks themselves can in turn be run in parallel. Thus, CI can be viewed as a mixed data-parallel application. A crucial point for a successful CI process is its ability to provide quick feedback. Thus, makespan minimization is the main goal. Our contribution is twofold. First we provide and analyze a large dataset corresponding to a build DAG. Second, we compare the performance of several scheduling heuristics on this dataset.

Automated software integration flows in industry: a multiple-case study

“…This prompted us to develop a descriptive model to capture these differences, allowing us to more accurately study, document and compare different variants. This model builds on the concept of integration as a Directed Acyclic Graph (DAG), as suggested by [6], with automated activities constituting interdependent nodes in that graph. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Scheduling Parallel Task Graphs on non-dedicated heterogeneous multicluster platform with Moldable Task Duplication

“…Real world workflow applications [5] executed on parallel computing platforms can exploit two types of parallelism: task parallelism and data parallelism [6]. As structured as Parallel Task Graphs (PTG), each data parallel task in the mixed-parallel workflow application is allocated a subset of the available processors, and several such tasks are executed concurrently in task-parallel fashion.…”