Optimizing the Data-Traffic of Centrally Coordinated Scientific Workflow Systems

Subramanian, S.; Puntervoll, Pål; Sztromwasser, Paweł

doi:10.1109/icws.2010.71

Cited by 5 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An empirical evaluation of the proposed PDFD vs published DFD given in Subramanian et al (2010Subramanian et al ( , 2012. The DFD approach is briefly described in Section 2.…”

Section: Pipelined Data-flowmentioning

confidence: 99%

“…In regular orchestration, data from one service is passed to the next service through the workflow coordinator. To optimize orchestrated workflow performance, in a previous paper (Subramanian et al, 2010) we proposed DFD, which allows data to flow directly from one service to the next. Furthermore, we applied DFD to SOAP Web services in a follow up paper (Subramanian et al, 2012).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pipelined data‐flow delegated orchestration for data‐intensive eScience workflows

Subramanian

Sztromwasser²,

Puntervoll³

et al. 2013

International Journal of Web Information Systems

Self Cite

View full text Add to dashboard Cite

Purpose -eScience workflows use orchestration for integrating and coordinating distributed and heterogeneous scientific resources, which are increasingly exposed as web services. The rate of growth of scientific data makes eScience workflows data-intensive, challenging existing workflow solutions. Efficient methods of handling large data in scientific workflows based on web services are needed. The purpse of this paper is to address this issue. Design/methodology/approach -In a previous paper the authors proposed Data-Flow Delegation (DFD) as a means to optimize orchestrated workflow performance, focusing on SOAP web services. To improve the performance further, they propose pipelined data-flow delegation (PDFD) for web service-based eScience workflows in this paper, by leveraging from the domain of parallel programming. Briefly, PDFD allows partitioning of large datasets into independent subsets that can be communicated in a pipelined manner. Findings -The results show that the PDFD improves the execution time of the workflow considerably and is capable of handling much larger data than the non-pipelined approach. Practical implications -Execution of a web service-based workflow hampered by the size of data can be facilitated or improved by using services supporting Pipelined Data-Flow Delegation. Originality/value -Contributions of this work include the proposed concept of combining pipelining and Data-Flow Delegation, an XML Schema supporting the PDFD communication between services, and the practical evaluation of the PDFD approach.

show abstract

“…An empirical evaluation of the proposed PDFD vs published DFD given in Subramanian et al (2010Subramanian et al ( , 2012. The DFD approach is briefly described in Section 2.…”

Section: Pipelined Data-flowmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Pipelined data‐flow delegated orchestration for data‐intensive eScience workflows

Subramanian

Sztromwasser²,

Puntervoll³

et al. 2013

International Journal of Web Information Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…In a Classical orchestration approach, the workflow is executed by moving all the data between services of the workflow via the orchestrator. This indirect movement of data between the services of the workflow decreases the workflow performance (Subramanian et al 2010). The performance decreases further when the orchestrated workflow has to handle big-data in it.…”

Section: Introductionmentioning

confidence: 99%

Reducing data transfer in big-data workflows: the computation-flow delegated approach

Nunes¹,

Deshpande

2019

J. of Data, Inf. and Manag.

View full text Add to dashboard Cite

Existing orchestrated bioinformatics workflow execution approaches necessitate the transfer of datasets from biological data services to the analysis tool (computation) services of the workflow for various data analysis. This model of moving data to computation during workflow execution weakens the performance of the workflow especially when the orchestrated bioinformatics workflow has to handle big-data in it. Since the size of the analysis tools are much smaller than the datasets size in a workflow, in this paper, to minimize the dataflow and improve workflow performance, we propose a novel computation-flow delegated (CFD) approach. The CFD approach lets the tool services of the workflow to dynamically migrate analysis tools towards the datasets to perform computation on data side during workflow execution. We use a set of mobile agents to operate the CFD approach and present a mobile agent-based computation-flow delegation framework (MABCFD) to execute the workflow tasks. We implement the prototype of the MABCFD framework and analyze the performance of the CFD approach empirically by executing in isolation workflow patterns (sequence, fan-out and fan-in) common to bioinformatics applications. Performance analysis shows that the computation-driven CFD approach consistently outperforms the existing data-driven approaches across all patterns and scales favorably with data size.

show abstract

“…The data-flow delegation happens on the fly according to the workflow requirements, i.e., the coordinator instructs the component services where the input data will be sent from, and where to send the output data. In this paper, we show that the data-flow delegation proposed in our previous paper [20] can be applied to delegate data between plain SOAP web services. We used concept of operation overloading to inject data-flow delegation instructions into service requests.…”

Section: Introductionmentioning

confidence: 99%

“…Mediating the large data-transfer between the component services increases use of the coordinator's resources excessively (e.g., memory, bandwidth), and increases the workflow execution time. In [20], we have proposed an approach called DataFlow Delegation to overcome this data-flow bottleneck. In that paper, we have given a theoretical foundation, for separating the control and the data-flow in a workflow execution in order to delegate the data-flow responsibilities to the component services.…”

Section: Introductionmentioning

confidence: 99%

Direct data transfer between SOAP web services in orchestration

Subramanian

Sztromwasser

Petersen

et al. 2012

Proceedings of the 14th International Conference on Information Integration and Web-Based Applications &Amp; Services

Self Cite

View full text Add to dashboard Cite

In scientific data analysis, workflows are used to integrate and coordinate resources such as databases and tools. Workflows are normally executed by an orchestrator that invokes component services and mediates data transport between them. Scientific data are frequently large, and brokering large data increases the load on the orchestrator and reduces workflow performance. To remedy this problem, we demonstrate how plain SOAP web services can be tailored to support direct service-to-service data transport, thus allowing the orchestrator to delegate the data-flow. We formally define a data-flow delegation message, develop an XML schema for it, and analyze performance improvement of data-flow delegation empirically in comparison with the regular orchestration using an example bioinformatics workflow.

show abstract

Optimizing the Data-Traffic of Centrally Coordinated Scientific Workflow Systems

Cited by 5 publications

References 9 publications

Pipelined data‐flow delegated orchestration for data‐intensive eScience workflows

Pipelined data‐flow delegated orchestration for data‐intensive eScience workflows

Reducing data transfer in big-data workflows: the computation-flow delegated approach

Direct data transfer between SOAP web services in orchestration

Contact Info

Product

Resources

About