An evaluation of the state of time synchronization on leadership class supercomputers

Abstract

SummaryWe present a detailed examination of time agreement characteristics for nodes within extreme-scale parallel computers. Using a software tool we introduce in this paper, we quantify attributes of clock skew among nodes in three representative high-performance computers sited at three national laboratories. Our measurements detail the statistical properties of time agreement among nodes and how time agreement drifts over typical application execution durations.We discuss the implications of our measurements, why the current state of the field is inadequate, and propose strategies to address observed shortcomings. KEYWORDSclock synchronization, large-scale systems, system software, time service INTRODUCTIONThe trend towards increasing node counts in high-performance computing (HPC) is motivating a move toward greater levels of concurrency in HPC systems. Today's software environment is now being called on to produce new solutions for emerging issues including managing system power, resilience, and performance characteristics. The distributed algorithms that underlie such services operate much more efficiently in the presence of tightly synchronized clocks. For example, tightly synchronized clocks benefit well-known gang scheduling techniques and complex consensus algorithms. To illustrate the point, such time synchronization enables more aggressive assumptions about communication and synchronization patterns, the removal of unnecessary locks, and a wide range of other applications. Clock-based techniques are already frequently deployed in cloud and data center distributed systems for precisely these reasons.We examined the time synchronization on some of the world's fastest and most powerful machines. These leadership-class systems employ high-end hardware connected by an extremely low-latency, low-jitter, interconnect in a carefully controlled environment, in contrast to widely distributed cloud-based systems based on commodity hardware and networks. Because of this, we assumed that these systems would have more stable, predictable hardware clocks, and close base time agreement using only standard time synchronization systems like Network Time Protocol (NTP). We did not believe that the complex hardware and software techniques used to provide time synchronization in wide-area systems would be necessary in leadership systems.Our results demonstrate that the actual time uncertainty for leadership-class machines is often unexpectedly large, in some cases over 600 milliseconds despite network latencies of less than two microseconds. Building on this, we set out to thoroughly quantify the magnitude of the time synchronization challenge in leadership-class systems. This study shows that the current time protocol in use, NTP, is not suitable for providing the level of time synchronization necessary for important system software tasks such as coordinated scheduling. Based on this, we conclude

Evaluation of NTP/PTP fine-grain synchronization performance in HPC clusters

“…Time synchronization is a key factor for any distributed system [13]. This is especially true for High Performance Computing (HPC) clusters, considering the trend toward increasing the node count to exploit application concurrency and parallelism.…”

“…This is especially true for High Performance Computing (HPC) clusters, considering the trend toward increasing the node count to exploit application concurrency and parallelism. In this context, it becomes crucial to ensure a tight level of time agreement [5,13,14]. As reported in [13], this is needed for several reasons, with the ultimate goal to improve the application execution time and energy efficiency of the system (i.e., Time-to-Solution -TtS -and Energy-to-Solution -EtS).…”

“…In this context, it becomes crucial to ensure a tight level of time agreement [5,13,14]. As reported in [13], this is needed for several reasons, with the ultimate goal to improve the application execution time and energy efficiency of the system (i.e., Time-to-Solution -TtS -and Energy-to-Solution -EtS). As example, a fine-grain synchronization (order of few µs) is fundamental in (i) wall-clock runtime of parallel applications that exploit the Bulk Synchronous Parallelism (BSP), which is a parallel programming model where the progress of concurrent processes is driven by synchronization points (synchronization barriers).…”

“…As example, a fine-grain synchronization (order of few µs) is fundamental in (i) wall-clock runtime of parallel applications that exploit the Bulk Synchronous Parallelism (BSP), which is a parallel programming model where the progress of concurrent processes is driven by synchronization points (synchronization barriers). As shown in [13], by mean of a coordinated scheduling (co-scheduling) strategy it is possible to achieve a speedup of 285 % for sensitive bulk synchronous workloads (i.e., with frequent synchronizing collectives based on Message Parallel Interface -MPI -, such as MPI_Barrier or MPI_Alltoall).…”

“…Other examples are (ii) code development / debugging and (iii) allow autotuning techniques to exploit new generation of high resolution monitoring systems [12,13,16]. Works in [4][5][6] show that synchronization performance in the order of few microseconds is a must to carry out an efficient analysis of applications' performance, as otherwise would not be possible to achieve a correct ordering of the events in concurrent processes running on multiple nodes.…”