Validation and mismatch repair of workflows through typed data streams

Abstract

The type system of a language guarantees that all of the operations on a set of data comply with the rules and conditions set by the language. While language typing is a fundamental requirement for any programming language, the typing of data that flow between processing elements within a workflow is currently being treated as optional. In this paper, we introduce a three-level type system for typing workflow data streams. These types are parts of the Data Intensive System Process Engineering Language programming language, which empowers users with the ability to validate the connections inside a workflow composition, and apply appropriate data type conversions when necessary. Furthermore, this system enables the enactment engine in carrying out type-directed workflow optimizations.

Data-intensive architecture for scientific knowledge discovery

“…The registry is also equipped with type propagation and checking functionalities that can be used by the application development tools or the enactment engines so as to verify that a workflow is well constructed (in terms of the types used in each connection) and to characterise the results of a workflow execution [38].…”

“…5: 1. DISPEL Language Processing, which includes parsing and validating a DISPEL program [7,38] and creating the data-flow graphs. 2.…”

Selected papers from the 2010 e-Science All Hands Meeting

“…Furthermore, such a performance database can be used to make decisions to optimize workflow execution. In a related paper, Yaikhom et al [4] discuss the DISPEL type system for workflow data streams, which can be used to validate the connections in a composite workflow, and apply data-type conversions, if needed. Workflow engines could also optimize DISPEL workflows by aggregating data transfers.…”

Performance database: capturing data for optimizing distributed streaming workflows

“…Site b comprises a gateway (DIVM g ), a data source (DIVM d ) and n ODEs ([DIVM k1 , DIVM k2 , …, DIVM kn ]). The Registry hosted in Site b keeps all the semantic information about the PEs and data sources that are used for type validation and optimization [6,7]. The type information includes the structure of the value streaming through a connection.…”

“…The type observer is applied during the Data-Intensive System Process Engineering Language (DISPEL) processing stage [7]. DISPEL is a language developed in the ADMIRE project which is processed to generate a graph for optimization and enactment.…”