Performance metrics and auditing framework using application kernels for high‐performance computer systems

Abstract

SUMMARYThis paper describes XSEDE Metrics on Demand, a comprehensive auditing framework for use by highperformance computing centers, which provides metrics regarding resource utilization, resource performance, and impact on scholarship and research. This role-based framework is designed to meet the following objectives: (1) provide the user community with a tool to manage their allocations and optimize their resource utilization; (2) provide operational staff with the ability to monitor and tune resource performance; (3) provide management with a tool to monitor utilization, user base, and performance of resources; and (4) provide metrics to help measure scientific impact. Although initially focused on the XSEDE program, XSEDE Metrics on Demand can be adapted to any high-performance computing environment. The framework includes a computationally lightweight application kernel auditing system that utilizes performance kernels to measure overall system performance. This allows continuous resource auditing to measure all aspects of system performance including filesystem performance, processor and memory performance, and network latency and bandwidth. Metrics that focus on scientific impact, such as publications, citations and external funding, will be included to help quantify the important role high-performance computing centers play in advancing research and scholarship.

Enabling comprehensive data-driven system management for large computational facilities

“…The XDMoD (XSEDE Metrics on Demand) tool is being developed to provide such a comprehensive resource management framework for XSEDE and high-performance computing centers [2]. To date the focus of XDMoD has been primarily on the development of usage, system performance, and scientific impact metrics.…”

“…The open source tools include: CLUMON [1], PCP [2], Ganglia [3], Nagios [4], NEWT [5], Lorenz [6], and SLURM [7]. Splunk [8] is a commonly used commercial system.…”

Time-bound analytic tasks on large datasets through dynamic configuration of workflows

“…Because our focus is on extending workflow systems to generate and use workflow performance models, our models are quite simple and can be extended in future work to incorporate techniques from this body of work. Workflow runtime is one of many metrics that must be taken into account, others can include resource utilization and reliability [Furlani et al 2013;Carrington et al 2005]. Predicting workflow performance is key for resource reservation and resource allocation.…”

“…To train a performance model for the topic modeling workflows discussed here, we used public datasets of document collections containing news items that are widely used in the natural language processing and machine learning communities 5 . The metadata extracted by OODT includes their sizes, which are as follows: R8_train: 3.2MB, R8_test: 1.1MB, R52_train: 4.1MB, R52_test: 1.5MB.…”

Automatically mining program build information via signature matching

“…This flag instructs the compiler to put each function in its own .text section instead of all functions from the same source file in one single .text section. So for a Fortran function, say foo, the compiler creates a section named .text.foo which consists of foo's code only 1 . In such a situation, our tool emits one signature for one such .text section.…”