An update on the scalability limits of the Condor batch system

Abstract

Condor is being used extensively in the HEP environment. It is the batch system of choice for many compute farms, including several WLCG Tier 1s, Tier 2s and Tier 3s. It is also the building block of one of the Grid pilot infrastructures, namely glideinWMS. As with any software, Condor does not scale indefinitely with the number of users and/or the number of resources being handled. In this paper we are presenting the current observed scalability limits of both the latest production and the latest development release of Condor, and compare them with the limits reported in previous publications. A description of what changes were introduced to remove the previous scalability limits are also presented.

AixViPMaP®—an Operational Platform for Microstructure Modeling Workflows

“…In the future, this allows to add not only local institutes but also organizations and their hardware across the globe. This includes, but is not limited to, scalability in terms of the number of addressable hardware and scalability in terms of the spatial extent of the network [41][42][43][44]. Table 2 lists a number of currently available grid middleware components.…”

Distributed scheduling and data sharing in late-binding overlays

“…There is however a major problem with this view: the central queue of tasks may become a bottleneck for the scheduling and execution of large-scale workloads [2], [8]. As the number of pilots increases, so does the rate of requests to be served.…”

“…Existing pilot systems have already met this problem and have found different ways to tackle it. Tasks may be grouped and evaluated in bulk or pilots may be matched based on their site only [8]. In this case, we are effectively restricting or even giving up micro-scheduling.…”

Sparrow

“…High performance computing schedulers fall into this category: they optimize for large jobs with complex constraints, and target maximum throughput in the tens to hundreds of scheduling decisions per second (e.g., SLURM [10]). Similarly, Condor supports complex features including a rich constraint language, job checkpointing, and gang scheduling using a heavy-weight matchmaking process that results in maximum scheduling throughput of 10 to 100 jobs per second [4].…”