Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

Abstract

Abstract-As high-end systems move toward exascale sizes, a new model of scientific inquiry being developed is one in which online data analytics run concurrently with the high end simulations producing data outputs. Goals are to gain rapid insights into the ongoing scientific processes, assess their scientific validity, and/or initiate corrective or supplementary actions by launching additional computations when needed. The Flexpath system presented in this paper addresses the fundamental problem of how to structure and efficiently implement the communications between high end simulations and concurrently running online data analytics, the latter comprised of componentized dynamic services and service pipelines.Using a type-based publish/subscribe approach, Flexpath encourages diversity by permitting analytics services to differ in their computational and scaling characteristics and even in their internal execution models. Flexpath uses direct and MxN connections between interacting services to reduce data movements, to allow for runtime connectivity changes to accommodate component arrivals/departures, and to support the multiple underlying communication protocols used for analytics workflows in which simulation outputs are processed by analytics services residing on the same nodes where they are generated, on the same machine, and/or on attached or remote analytics engines. This paper describes the design and implementation of Flexpath, and evaluates it with two widely used scientific applications and their associated data analytics methods.

Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures

“…ADIOS [18,19] is an adaptive I/O service that is designed to allow applications to easily change between different I/O service providers. Only a tweak to the input parameters is needed to swap methods.…”

“…The FlexPath transport [19] within ADIOS offers the ability to configure a set of resources for doing in situ computations in a number of configurations: on node (traditional in situ), across several nodes (in transit), or in hybrid combinations. For this paper, we study the performance of the Histogram, Autocorrelation, and Catalyst-slice configurations using an ADIOS FlexPath endpoint (as in Figure 2).…”

The SENSEI Generic In Situ Interface

“…This is the approach taken by ADIOS [12], [5] and GLEAN [13]. This can be used to support relatively complex in situ deployments that can adapt their pipeline on the fly based on resource issues [14].…”

“…The Execute function allows ADIOS to then transmit data using one of several transports that support in situ; our current work has been focused on the FlexPath transport [5]. Because of the adaptor + endpoint strategy, the ADIOS method can be used to transport data to a different staging area (on node or off), where LibSim or Catalyst can then be invoked as an in situ function within the endpoint executable as a composite workflow.…”

Visualization and Analysis for Near-Real-Time Decision Making in Distributed Workflows

“…SYSTEM IMPLEMENTATION Our visualization system is spread across two different geographic locations, The A*STAR Computational Resource Centre in Singapore and Georgia Tech in Atlanta, and is a synthesis of a variety of different tools and frameworks. On a high level this system consists of the simulation and data manipulation routines located in Singapore, the interactive visualization and analysis routines located at Georgia Tech, and a connection over the wide area network using the ADIOS [8] middleware and the the ICEE [9], FlexPath [10], and DataSpaces [11] transport methods. Combined, these technologies allowed us to perform memory to memory data delivery from one side of the pipeline to the other.…”