Integrating databases and workflow systems

Abstract

There has been an information explosion in fields of science such as high energy physics, astronomy, environmental sciences and biology. There is a critical need for automated systems to manage scientific applications and data. Database technology is well-suited to handle several aspects of workflow management. Contemporary workflow systems are built from multiple, separately developed components and do not exploit the full power of DBMSs in handling data of large magnitudes. We advocate a holistic view of a WFMS that includes not only workflow modeling but planning, scheduling, data management and cluster management. Thus, it is worthwhile to explore the ways in which databases can be augmented to manage workflows in addition to data. We present a language for modeling workflows that is tightly integrated with SQL. Each scientific program in a workflow is associated with an active table or view. The definition of data products is in relational format, and invocation of programs and querying is done in SQL. The tight coupling between workflow management and data-manipulation is an advantage for data-intensive scientific programs.

Querying an Integrated Complex-Object Dataflow Database

“…In order to be able to organize the data flow, we also want the basic operations on tuples and sets: tuple formation, tuple projection, singleton set formation, set union, and big union, also known as "flatten". 3 Finally, we need an if-then-else construct. This rounds up the operations of a natural language for complex objects known as the nested relational calculus or NRC.…”

“…Workflow management has its origins in business process modeling [1], but in recent years workflows have gained importance in e-science, in parallel with the rise of Grid Computing [2]. Scientific workflows are distinguished from business workflows by their placing more importance on the data flow between the subtasks, than on the synchronization of subtasks [3]. (In escience, the data flow frequently involves collections of complex data objects.)…”

A Formal Model of Dataflow Repositories

“…Several researchers advocate integration of workflows and DBMSs [9,10,11], as they provide mechanisms for planning, scheduling, and logging. We believe that to properly design a dataflow repository, you need a formal model for dataflows and runs.…”

“…OPM is implemented in RDBMS, and runs are stored as instances of relational schemas. More recently, Shankar et al [11] have proposed dataflow specification integrated with SQL and relational DBMS. Dataflows are modeled as active relational tables, and invoked through SQL queries.…”

“…An approach that seems promising to us, and which is the object of our current work, is to build the system on top of a modern relational DBMS using SQL/PSM and SQL/XML. A similar approach was also advocated by Shankar et al [11]. Base values are implemented using SQL datatypes; more complicated base types such as NCBIXML can be implemented using the XML column type, or as large objects (LOBs).…”