Using XDMoD to facilitate XSEDE operations, planning and analysis

Abstract

The XDMoD auditing tool provides, for the first time, a comprehensive tool to measure both utilization and performance of high-end cyberinfrastructure (CI), with initial focus on XSEDE. Here, we demonstrate, through several case studies, its utility for providing important metrics regarding resource utilization and performance of TeraGrid/XSEDE that can be used for detailed analysis and planning as well as improving operational efficiency and performance. Measuring the utilization of high-end cyberinfrastructure such as XSEDE helps provide a detailed understanding of how a given CI resource is being utilized and can lead to improved performance of the resource in terms of job throughput or any number of desired job characteristics. In the case studies considered here, a detailed historical analysis of XSEDE usage data using XDMoD clearly demonstrates the tremendous growth in the number of users, overall usage, and scale of the simulations routinely carried out. Not surprisingly, physics, chemistry, and the engineering disciplines are shown to be heavy users of the resources. However, as the data clearly show, molecular biosciences are now a significant and growing user of XSEDE resources, accounting for more than 20 percent of all SUs consumed in 2012. XDMoD shows that the resources required by the various scientific disciplines are very different. Physics, Astronomical sciences, and Atmospheric sciences tend to solve large problems requiring many cores. Molecular biosciences applications on the other hand, require many cycles but do not employ core counts that are as large. Such distinctions are important in guiding future cyberinfrastructure design decisions. XDMoD's implementation of a novel application kernel-based auditing system to measure overall CI system performance and quality of service is shown, through several examples, to provide a useful means to automatically detect under performing hardware and software. This capability is especially critical given the complex composition of today's advanced CI. Examples include an application kernel based on a widely used quantum chemistry program that uncovered a software bug in the I/O stack of a commercial parallel file system, which was subsequently fixed by the vendor in the form of a software patch that is now part of their standard release. This error, which resulted in dramatically increased execution times as well as outright job failure, would likely have gone unnoticed for sometime and was only uncovered as a result of implementation of XDMoD's suite of application kernels.

XD Metrics on Demand Value Analytics: Visualizing the Impact of Internal Information Technology Investments on External Funding, Publications, and Collaboration Networks

“…The results revealed that PIs constituted 23% (3,334) of the total TeraGrid project users (14,474) and that molecular biosciences and chemistry are the two scientific fields with the highest number of projects involving TeraGrid usage, 2,292 and 1,828, respectively. Similarly, Furlani et al (2012Furlani et al ( , 2013a observed that molecular biosciences recently joined physics at the top of the list of sciences with the greatest CPU usage of Extreme Science and Engineering Discovery Environment (XSEDE), a virtual system providing digital resources and computing services (Towns et al, 2014). In addition, Furlani's longitudinal study demonstrated a substantial increase in the number of PIs using XSEDE from ~500 in 2005 to ~1,600 in 2011.…”

Application kernels: HPC resources performance monitoring and variance analysis

“…Application kernel process control. 5 Process the output if it was collected. Indeed, the Copy it to HPC resource and submit to queue.…”

“…Here, the IOR application kernel clearly shows a dramatic and sudden drop-off in write performance in January 2014. Additional application kernel success stories include (1) detection of a 25% degradation of NAMD performance after an 'upgrade' [3]; (2) detection of a software bug in the I/O stack of a commercial parallel file system that caused sporadic behavior in NWChem [5]; and (3) a sudden decrease in file system performance on TACC Lonestar4 as measured by three different application kernels (IOR, MPI-Tile-IO, and IMB) [6]. As shown in Figure 9, the file system performance was eventually returned to normal after working closely with the switch manufacturer and the vendor of the parallel file system.…”

“…Here, a brief overview of XD metrics on demand (XDMoD) is presented; a more detailed description can be found in references [1][2][3][4][5][6]. XDMoD was originally designed to provide the analyses required for effective overall management of the computational and operational resources of the Extreme Science and Engineering Discovery Environment (XSEDE) [7] as part of the XSEDE Technology Audit Service.…”

Accessing multiple clouds with cloudmesh

“…However, for this paper we will restrict our discussion to the IaaS based monitoring components. Other HPC related activities in regards to monitoring and metrics are discussed in [4].…”