Aggregation of Real-Time System Monitoring Data for Analyzing Large-Scale Parallel and Distributed Computing Environments

Abstract

We present a monitoring system for large-scale parallel and distributed computing environments that allows to trade-off accuracy in a tunable fashion to gain scalability without compromising fidelity. The approach relies on classifying each gathered monitoring metric based on individual needs and on aggregating messages containing classes of individual monitoring metrics using a tree-based overlay network. The MRNet-based prototype is able to significantly reduce the amount of gathered and stored monitoring data, e.g., by a factor of ≈56 in comparison to the Ganglia distributed monitoring system. A simple scaling study reveals, however, that further efforts are needed in reducing the amount of data to monitor future-generation extreme-scale systems with up to 1,000,000 nodes. The implemented solution did not had a measurable performance impact as the 32-node test system did not produce enough monitoring data to interfere with running applications.

Resilient gossip algorithms for collecting online management information in exascale clusters

“…It has been used for scalable parallel performance-monitoring and profiling of high-performance computing applications [8,20]. MRNet can pass filtered information up and down the tree, which can be processed at each tree node.…”

“…This size is slightly larger than the 64 bytes used by Bohm et al [8] for monitoring and the 84 bytes used in MOSIX [9], a decentralized cluster management system that uses process migration for load balancing. Recall that the global information must also be included.…”

“…A reasonable local entry size is 100 bytes. This size is slightly larger than the 64 bytes used by Bohm et al [8] for monitoring and the 84 bytes used in MOSIX [9], a decentralized cluster management system that uses process migration for load balancing. MOSIX clusters resemble a single colony without a master, where each entry includes a node's CPU and memory information and status.…”

“…This paper presents a new paradigm for providing online information to the management system of scalable clusters, consisting of a large number of nodes and one or more masters that manage these nodes. It has been used for scalable parallel performance-monitoring and profiling of high-performance computing applications [8,20]. The presented algorithms are decentralized, scalable and resilient, working well even when some nodes fail, without needing any recovery protocol.…”

“…MRNet can pass filtered information up and down the tree, which can be processed at each tree node. It has been used for scalable parallel performance-monitoring and profiling of high-performance computing applications [8,20]. However, MRNet is unsuitable for passing information between nodes of the same tree level, as required by our colony's internal management system, because it requires messages to pass through a common parent, rather than directly.Cluster management systems are known to perform well on moderate-sized Linux clusters, but the management overhead on large-scale supercomputers is not well understood, as these systems are more susceptible to network contentions.…”