A modeling and execution environment for distributed scientific workflows

Scientific Data Management Integrated Software Infrastructure Center (SDM/ISIC): Scientific Process Automation (SPA), FINAL REPORT

“…The open source Kepler project started in 2003 when members of the SciDAC-SDM Center/SPA Team, sponsored by this DOEfunded project (DE-FC02-01ER25486) and members of the SEEK project (NSF/ITR awards DBI-0225674 and DBI-0533368) decided to collaborate and jointly develop a scientific workflow system based on the open source Ptolemy ii system (Ludäscher was a co-PI on both projects). In the first project phase, between 2001 and 2003, the SDSC team (Altintas, Ludäscher) worked closely with a domain scientist (Matt Coleman, LLNL) and created early versions of a Promoter Identification Workflow (PIW) [15,62]. Towards the end of that period, the open source Ptolemy ii system was adopted by SciDAC-SDM and SEEK as the basis for a general scientific workflow system and problem-solving environment to design and execute scientific workflows, giving rise to Kepler.…”

“…We take a closer look at underlying technical issues and challenges in Section 4. Figure 1 shows a high-level, conceptual view of a typical scientific knowledge discovery workflow that links genomic biology techniques such as microarrays with bioinformatics tools such as BLAST to identify and [15] characterize eukaryotic promoters 5 -we call this the Promoter Identification Workflow or PIW (see also [79,15,62]: Starting from microarray data, cluster analysis algorithms are used to identify genes that share similar patterns of gene expression profiles that are then predicted to be co-regulated as part of an interactive biochemical pathway. Given the gene-ids, gene sequences are retrieved from a remote database (e.g., GenBank) and fed to a tool (e.g., BLAST) that finds similar sequences.…”

Scalable peer-to-peer process management - the OSIRIS approach

“…Albeit navigation tasks can be distributed in a cluster, storage of process instances usually is done by using a single, centralized database instance (products like Oracle 10g [24] can also support clustered databases). Some prototype systems features similar architectures (e.g., Mentorlite [16], SDM [2]). Although executing processes in a peer-to-peer way without involving a centralized component, OSIRIS provides transactional guarantees, following the model of transactional processes [30].…”

Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis

“…Scientific workflow systems have been used for accessing data from a variety of sources, including database systems (Altintas et al, 2004a), Grid systems (Altintas et al, 2003;Altintas et al, 2005;Deelman et al, 2005;Ludäscher et al, 2006;Taylor et al, 2007b), and Web Services (Altintas et al, 2004b). In addition, the Kepler workflow system has built-in tools for accessing heterogeneous environmental data by using details about data content and structure from metadata descriptions available in the Knowledge Network for Biocomplexity (KNB), a large-scale, distributed data system.…”