Efficient Runtime Capture of Multiworkflow Data Using Provenance

Abstract

Computational Science and Engineering (CSE) projects are typically developed by multidisciplinary teams. Despite being part of the same project, each team manages its own workflows, using specific execution environments and data processing tools. Analyzing the data processed by all workflows globally is a core task in a CSE project. However, this analysis is hard because the data generated by these workflows are not integrated. In addition, since these workflows may take a long time to execute, data analysis needs to be done at runtime to reduce cost and time of the CSE project. A typical solution in scientific data analysis is to capture and relate the data in a provenance database while the workflows run, thus allowing for data analysis at runtime. However, the main problem is that such data capture competes with the running workflows, adding significant overhead to their execution. To mitigate this problem, we introduce in this paper a system called ProvLake, which adopts design principles for providing efficient distributed data capture from the workflows. While capturing the data, ProvLake logically integrates and ingests them into a provenance database ready for analyses at runtime. We validated ProvLake in a real use case in the O&G industry encompassing four workflows that process 5 TB datasets for a deep learning classifier. Compared with Komadu, the closest solution that meets our goals, our approach enables runtime multiworkflow data analysis with much smaller overhead, such as 0.1%. Geological raw data files Kubernetes VolumeInter-workflow Data RelationshipsMulti-store Data Relationships Legend Deep learning training datasets

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

“…Users can use online provenance analysis to monitor, debug or inspect the data transformations while they are still running (e.g., see the status, see how the intermediate results are evolving as the input parameters vary). The problem of adding low provenance data capture overhead is more challenging for provenance systems that allow for online analysis [17]. Queries Q3-Q5 exemplify queries that can be executed online, e.g., while a training process is running.…”

“…Provenance tracking comprises provenance capture, the creation of the provenance relationships (e.g., associations between the processes and the consumed and generated data), and storage of the provenance data. In our view, provenance tracking systems that can be coupled to workflows [12,13,15,17] provide the flexibility needed in large-scale CSE projects, as opposed to moving workflows' executions and data to be managed by a single orchestration system, like a Workflow Management System. Workflow provenance capture systems usually address scripts as workflows with chained functions, method, or library calls that execute data transformations, while capturing input arguments and output values from these calls.…”

“…The workflows are instrumented with PLLib, imported as a library in the scripts, which is responsible for the provenance data capture. In an offline manner and following the methodology described in a previous work [17] to specify the workflows using prospective provenance data standards [8], users add provenance data capture calls using the PLLib. A provenance capture task happens when a data transformation executes, which typically occurs in a function call, in a…”

“…Because of the workflow specification using prospective provenance data, kept external to the workflow scripts and loaded only once at the beginning (Line 2 in Algorithm 1), the code modification and data to be sent to ProvTracker are reduced. Design principles such as queuing provenance requests, asynchronicity (i.e., the workflow scripts do not wait for the provenance requests to be fully processed-the pipeline from the PLLib to ProvTracker, ProvManager, and Prov DBMS), and reduction of system calls help reducing capture overhead [17]. Provenance capture requests are queued and the maximum queue size is a configurable parameter.…”

“…Nonetheless, solutions following this approach do not support the lifecycle, which requires three main capabilities: provenance tracking in multiple workflows that use heterogeneous data and stores; a provenance data representation with ML-specific vocabulary; and providing for integrated data analysis through provenance while keeping the capture overhead low. Another solution following this approach is ProvLake [17], which has low capture overhead in multiple workflows that use heterogeneous data stores. However, similarly to the other solutions, its provenance data representation is based on W3C PROV only, thus lacking ML-specific vocabulary, limiting its support for the ML lifecycle in CSE.…”