Chimbuko: A Workflow-Level Scalable Performance Trace Analysis Tool

Abstract

Due to the sheer volume of data it is typically impractical to analyze the detailed performance of an HPC application running at-scale. While conventional small-scale benchmarking and scaling studies are often sufficient for simple applications, many modern workflow-based applications couple multiple elements with competing resource demands and complex inter-communication patterns for which performance cannot easily be studied in isolation and at small scale. This work discusses Chimbuko, a performance analysis framework that provides real-time, in situ anomaly detection. By focusing specifically on performance anomalies and their origin (aka provenance), data volumes are dramatically reduced without losing necessary details. To the best of our knowledge, Chimbuko is the first online, distributed, and scalable workflow-level performance trace analysis framework. We demonstrate the tool's usefulness on Oak Ridge National Laboratory's Summit system. CCS CONCEPTS • Software and its engineering → Software creation and management; • Human-centered computing → Visualization techniques; • Computing methodologies → Parallel computing methodologies.

SOMA: Observability, monitoring, and in situ analytics for exascale applications

“…Tools like Falcon, 33 Autopilot, 34 Periscope, 35 ActiveHarmony, 36 and WOWMON 37 utilized monitoring to provide online analysis and support for adapting and steering the application. Chimbuko, 38 utilizing TAU as a performance measurement system, implemented in situ trace analysis to detect performance anomalies and generate provenance for root cause analysis.…”

HEPnOS: a Specialized Data Service for High Energy Physics Analysis

“…DataSpaces [23], for example, provides a N-dimentional data model for coupling parallel applications in workflows. Chumbuko [24] and SEER [25] respectively use a documentstorage and a key/value-storage model to store performance data. Services like DAOS [26] provide a distributed file-system interface on top of storage-class memory, but also a lowerlevel object-store interface that applications can tailor to their particular needs.…”

Traveler: Navigating Task Parallel Traces for Performance Analysis

“…Sanderson et al [34] show streaming data for performance analysis and steering, with statistical plots and the projecting of metrics onto the 3D simulation space and machine room layout. Chuimbuko [21] also shows streaming data, with statistical plots and a call stack plot of streaming data. Of these, only Chimbuko's call stack plot shows dependencies and like SyncTrace it is limited to a specific time range and focus resource.…”