The Influence of Operating Systems on the Performance of Collective Operations at Extreme Scale

Beckman, Pete; Iskra, Kamil; Yoshii, Kazutomo; Coghlan, Susan

doi:10.1109/clustr.2006.311846

Cited by 69 publications

(51 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ferreira, Bridges, and Brightwell use noise-injection techniques to assess the impact of noise on several applications [5]. Beckman et al [6] analyzed the performance on BlueGene/L, concluding that most sources of noise can be avoided in very specialized systems.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation

Hoefler

Schneider

Lumsdaine

2010

2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

191

137

View full text Add to dashboard Cite

Abstract-This paper presents an in-depth analysis of the impact of system noise on large-scale parallel application performance in realistic settings. Our analytical model shows that not only collective operations but also point-to-point communications influence the application's sensitivity to noise. We present a simulation toolchain that injects noise delays from traces gathered on common large-scale architectures into a LogGPS simulation and allows new insights into the scaling of applications in noisy environments. We investigate collective operations with up to 1 million processes and three applications (Sweep3D, AMG, and POP) with up to 32,000 processes. We show that the scale at which noise becomes a bottleneck is system-specific and depends on the structure of the noise. Simulations with different network speeds show that a 10x faster network does not improve application scalability. We quantify noise and conclude that our tools can be utilized to tune the noise signatures of a specific system. I. MOTIVATION AND BACKGROUNDThe performance impact of operating system and architectural overheads (system noise) at massive scale is increasingly of concern. Even small local delays on compute nodes, which can be caused by interrupts, operating system daemons, or even cache or page misses, can affect global application performance significantly [1]. Such local delays often cause less than 1% overhead per process but severe performance losses can occur if noise is propagated (amplified) through communication or global synchronization. Previous analyses generally assume that the performance impact of system noise grows at scale and Tsafrir et al. [2] even suggest that the impact of very low frequency noise scales linearly with the system size. A. Related WorkPetrini, Kerbyson, and Pakin [1] report that the parallel performance of SAGE on a fixed number of ASCI Q nodes was highest when SAGE used only three of the four CPUs per node. It turned out that "resonance" between the application's collective communication and the misconfigured system caused delays during each iteration. Jones, Brenner, and Fier [3] observed similar effects with collective communication and also report that, under certain circumstances, it is beneficial to leave one CPU idle. A theoretical analysis of the influence of noise on collective communication [4] suggests that the impact of noise depends on the type of distribution and their parameters and can, in the worst case (exponential distribution), scale linearly with the number of processes. Ferreira, Bridges, and Brightwell use noise-injection techniques to assess the impact of noise on several applications [5]. Beckman et al.[6] analyzed the performance on BlueGene/L, concluding that most sources of noise can be avoided in very specialized systems.Previous work was either limited to experimental analysis on specific architectures with injection of artificially generated noise (fixed frequency), or to purely theoretical analyses that assume a particular collective pattern [4]. These previou...

show abstract

Section: A Related Workmentioning

confidence: 99%

“…For our experiments, we choose a FWQ benchmark with a workload close to zero. 1 To manage the huge number of measurements, we only store the time of the perturbed measurements similar to Beckman's "selfish detour" benchmark [6] which also uses a tight loop to measure perturbations.…”

Section: Measuring System Noisementioning

confidence: 99%

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation

Hoefler

Schneider

Lumsdaine

2010

2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis

191

137

View full text Add to dashboard Cite

show abstract

“…3 Notice that this step is required to perform kernel activities on the CPUs that did not received the interrupt from the network card.…”

Section: B Implementation Of Nettickmentioning

confidence: 99%

“…In [19] the authors observed that the impact of the system noise when scaling on 8,192 processors was so large (because of the so-called "noise resonance") that leaving one processor idle to take care of the system activities lead to a performance improvement of 1.87x. The impact of the operating system on classical MPI operations, such as collective, was examined in [3]. Though in [19] the authors did not identify every single source of OS noise, a following paper [11] identified timer interrupts (and the activities started by the paired interrupt handler) as the main source of OS noise.…”

Section: Related Workmentioning

confidence: 99%

A global operating system for HPC clusters

Betti

Cesati

Gioiosa

et al. 2009

2009 IEEE International Conference on Cluster Computing and Workshops

View full text Add to dashboard Cite

Abstract-Modern supercomputers consist of clusters of thousands of independent nodes interconnected through fast networks. These nodes run independent operating system kernels, thus synchronization among them is demanded for user mode programs. This means that temporal synchronization of the nodes is a daunting task. On the other hand, HPC cluster applications often require a rather strict temporal synchronization for activities like performance analysis, application debugging, or data checkpointing. Therefore, the performance of an HPC parallel application may be severely impaired by the lack of temporal synchronization among the activities of the nodes of the cluster; this poses a severe limit on the scalability of such architectures. In this paper we introduce CAOS, an extension of the Linux kernel that aims to address the temporal synchronization problems of modern HPC clusters. We describe the general ideas behind CAOS, and we discuss some details of a possible implementation. We also illustrate some experiments performed on a prototype implementation of CAOS including a centralized network time tick, which allows a master node to synchronize the activities of all other nodes in the cluster, and a specific task scheduler tailored for HPC applications. These experiments, performed on a modern HPC cluster, witness that this new component has no measurable impact on the efficiency of the nodes while reducing the OS noise and providing better performance prediction. An implementation of CAOS based on this component can achieve a significant gain in terms of synchronization, global control, and scalability of the cluster.

show abstract

“…Fedora Core). Note that this Linux distribution may not even be tuned for HPC, enabling several system services, which at the end create a important operating system noise, critical overhead in large-scale systems [22,4].…”

Section: Hypervisor For High Performance Computingmentioning

confidence: 99%

System-Level Virtualization for High Performance Computing

Vallée

Naughton

Engelmann

et al. 2008

16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008)

View full text Add to dashboard Cite

System-level virtualization has been a research topic since the 70's but regained popularity during the past few years because of the availability of efficient solution such as Xen and the implementation of hardware support in commodity processors (e.g. Intel-VT, AMD-V).However, a majority of system-level virtualization projects is guided by the server consolidation market. As a result, current virtualization solutions appear to not be suitable for high performance computing (HPC) which is typically based on large-scale systems. On another hand there is significant interest in exploiting virtual machines (VMs) within HPC for a number of other reasons. By virtualizing the machine, one is able to run a variety of operating systems and environments as needed by the applications. Virtualization allows users to isolate workloads, improving security and reliability. It is also possible to support nonnative environments and/or legacy operating environments through virtualization. In addition, it is possible to balance work loads, use migration techniques to relocate applications from failing machines, and isolate fault systems for repair.This document presents the challenges for the implementation of a system-level virtualization solution for HPC. It also presents a brief survey of the different approaches and

show abstract

The Influence of Operating Systems on the Performance of Collective Operations at Extreme Scale

Cited by 69 publications

References 10 publications

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation

A global operating system for HPC clusters

System-Level Virtualization for High Performance Computing

Contact Info

Product

Resources

About