I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes

Abstract

Lack of I/O scalability is known to cause measurable slowdowns for large-scale scientific applications running on high end machines. This is prompting researchers to devise 'I/O staging' methods in which outputs are processed via online analysis and visualization methods to support desired science outcomes. Organized as online workflows and carried out in I/O pipelines, these analysis components run concurrently with science simulations, often using a smaller set of nodes on the high end machine termed 'staging areas'. This paper presents a new approach to dealing with several challenges arising for such online analytics, including: how to efficiently run multiple analytics components on staging area resources providing them with the levels of end-to-end performance they need and how to manage staging resources when analytics actions change due to user or data-dependent behavior. Our approach designs and implements middleware constructs that delineate and manage I/O pipeline resources called 'I/O Containers'. Experimental evaluations of containers with realistic scientific applications demonstrate the feasibility and utility of the approach.

Efficient, Failure Resilient Transactions for Parallel and Distributed Computing

“…The transaction protocol helps manage resizing the resource allocation of the various analysis and visualization components. This has been described and analyzed previously [5]. This additional use for D 2 T demonstrates its generality.…”

Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

“…It is blocking issues in scenarios like these that motivate the notion of I/O Containers for managing analytics pipelines presented in [10]. Figure 5 depicts the overall improvements in runtimes for the LAMMPS and GTS applications.…”

“…Sample management constructs built in our own previous work [10], for example, have balanced pipeline operations to ensure QoS and have implemented transactional constructs with the goal of providing ACID properties for select online analytics [11], [12].…”

“…Instead, it makes possible the efficient realization of alternative communication scheduling techniques [13] and/or higher level methods for managing entire analytics workflows [10]. In contrast to web-based or commercial data streaming infrastructures [17], [21], [22], it does not constrain end users in how to write their analytics routines, so that they can leverage the rich tools already existing for these purposes, like R or MatLab.…”

ObsCon: Integrated Monitoring and Control for Parallel, Real-Time Applications

“…Real-time stream processing in the enterprise [1] [2] [3] and scientific [4] domains has become important to a wide range of applications, many of which require strong Quality of Service (QoS) guarantees for completing operations within application-specific end-to-end delays [5]. Such guarantees must be made in the presence of innate application dynamics, such as those caused by input-dependent data processing delays [6] and/or runtime failure [7].…”