The Impact of Noise on the Scaling of Collectives: A Theoretical Approach

Abstract

Abstract. The performance of parallel applications running on large clusters is known to degrade due to the interference of kernel and daemon activities on individual nodes, often referred to as noise. In this paper, we focus on an important class of parallel applications, which repeatedly perform computation, followed by a collective operation such as a barrier. We model this theoretically and demonstrate, in a rigorous way, the effect of noise on the scalability of such applications. We study three natural and important classes of noise distributions: The exponential distribution, the heavy-tailed distribution, and the Bernoulli distribution. We show that the systems scale well in the presence of exponential noise, but the performance goes down drastically in the presence of heavy-tailed of Bernoulli noise.

Impact of Noise on Scaling of Collectives: An Empirical Evaluation

“…It is increasingly becoming evident that operating system interference in the form of daemon activity and interrupts contribute significantly to performance degradation of parallel applications in large clusters. An earlier theoretical study has evaluated the impact of system noise on application performance for different noise distributions [1]. Our work complements the theoretical analysis by presenting an empirical study of noise in production clusters.…”

“…A formal approach to study the impact of noise in these large systems was initiated by Agarwal et al [1]. The parallel application studied was a typical class of kernel that appears in most scientific applications.…”

“…The theoretical model for capturing the impact of noise on cluster performance, based on [1], is presented in Section 2. In Section 3, we present the details of the parallel benchmark that we have designed.…”

“…We designed a parallel benchmark that was used on large clusters at SanDeigo Supercomputing Center for collecting noise related data. This data was fed to a simulator that predicts the performance of collective operations using the model of [1]. We report our comparison of the predicted and the observed performance.…”

A Case for Non-blocking Collective Operations

“…However, results can increase the application running time dramatically, as shown in [16]. Theoretical [17] and practical analyses [18,16] show that operating system noise and resulting process skew is definitively influencing the performance of parallel applications using blocking collective operations. Non-blocking collective operations avoid explicit synchronization unless it is necessary (if the programmer wants to wait for the operation to finish).…”