The Dynamics of Backfilling: Solving the Mystery of Why Increased Inaccuracy May Help

Abstract

Parallel job scheduling with backfilling requires users to provide runtime estimates, used by the scheduler to better pack the jobs. Studies of the impact of such estimates on performance have modeled them using a "badness factor" f ≥ 0 in an attempt to capture their inaccuracy (given a runtime r, the estimate is uniformly distributed in. Surprisingly, inaccurate estimates (f > 0) yield better performance than accurate ones (f = 0).We explain this by a "heel and toe" dynamics that, with f > 0, cause backfilling to approximate shortest-job first scheduling. We further find the effect of systematically increasing f is V-shaped: average wait time and slowdown initially drop, only to rise again later on. This happens because higher f s lead to increased randomness (more long jobs appear as short and vice versa) and to backfilling longer jobs, overshadowing the "heel and toe" dynamics and limiting the preference for short jobs. Finally, we show that the badness factor fails to capture the badness of real estimates, because these are modal and bounded by a maximal value.

Scheduling Restartable Jobs with Short Test Runs

“…Accounts initially differed on whether overestimates improved [10,18] or hurt [1] performance. These observations were eventually reconciled with the finding that overestimates initially help but that extreme overestimates eventually hurt performance [15,17]. Regardless of this, it seems reasonable that good estimates could be useful since they provide more information to the scheduler.…”

Workload Modeling for Computer Systems Performance Evaluation

“…Nevertheless, studies of user estimates reveal that they are often highly inaccurate, and may represent an overestimate by a full order of magnitude [506,430]. Surprisingly, this can sway the conclusions when comparing schedulers that use the estimates to decide whether to backfill jobs (that is, to use them to fill holes in an existing schedule) [237,696]. Thus assuming accurate estimates may lead to misleading evaluations [692].…”

“…The effect of such inaccurate estimates was investigated using a model where the real time is multiplied by a random factor from a limited range; the larger the range, the less accurate the resulting estimates. However, this model still retained information regarding the relative length of each job, and therefore enabled schedulers to achieve better results than they would in reality [696]. Thus, Tsafrir et al developed an intricate model to mimic the real relationship between job runtimes and user estimates [693].…”

“…This led to a line of interesting experimental work that eventually showed that inaccurate estimates cause holes to be left in the schedule, and these can then be filled with short jobs, effectively leading to an implicit shortest-job-first effect that improves the average response time. Thus the truth of the matter is that accurate estimates are actually better, because they facilitate using shortest-job-first explicitly [242,506,763,696,694]. The original study was also tainted by using "statistically bad" estimates that, as noted earlier, still contain significant information.…”

Evaluating the impact of inaccurate information in utility-based scheduling

“…Uncertainty exists, for example, when a user is unsure how to assign a value to a job (e.g., due to insufficient information about future demand or job importance [21,28,30]), or is otherwise unable to accurately determine job-specific characteristics prior to submission, such as estimated running time [16,22,32,33]. Marketinspired utility schedulers use various pricing mechanisms to extract truthful user valuations.…”