Big Provenance Stream Processing for Data Intensive Computations

Abstract

This dissertation is a result of an effort over many years. There are so many people who helped me in various ways during this endeavor. Without their generous support and encouragement, this work would not have been possible. First of all, I am so grateful to my Ph.D. advisor, Prof. Beth Plale for her invaluable support, guidance, and encouragement throughout my Ph.D. Her research experience over many years across multiple areas of Computer Science helped me in many ways to solve hard research problems and to successfully present them as publications. In addition to that, she was so kind to me and my family during our hard times. I am truly honored to have worked with her throughout my Ph.D. studies. I would like to thank my research committee members Prof. David Leake, Prof. Ryan Newton and Prof. Judy Qiu for their guidance and advice on my qualifying exams, thesis proposal, and final dissertation. I should thank all professors at the School of Informatics, Computing, and Engineering from whom I took a number of courses which helped immensely to improve my knowledge and skills.

Efficient Runtime Capture of Multiworkflow Data Using Provenance

“…Runtime multiworkflow provenance data capture. Komadu [8,12] is the only solution we found in this group. Different from the previous groups, Komadu aims at generating integrated provenance data as a multiworkflow runs.…”

“…To capture the data relationships between data transformations and workflows, which are given by the consumed and generated data values, the service uses unique identifiers to every data value that flows through it. Using unique identifiers for maintaining relationships of captured data is used in several provenance systems [8,12]. Thus, every data value receives a unique identifier in the PLView.…”

“…In Komadu, performance issues related to runtime data capture are left for future work [8], since it integrates the provenance graphs for queries only after all workflow executions end. Finally, a limitation found in all related work [4], [6]- [12] is the lack of query support from workflow data stored in heterogeneous databases.…”

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

“…It is unrealistic to expect that all phases, their execution, and the processed data will be managed by one single system. Alternatively, provenance tracking systems [12][13][14][15][16] can be coupled to a CSE workflow, providing provenance support while not significantly changing the way CSE users develop their applications. However, these solutions fail to track the interconnections between workflows and fail to track data processed in multiple heterogeneous stores.…”

“…11prov.out( batch , loss value)12 prov.out( epoch , conf usion matrix, model hyperprms, model perf , model ref )…”