A programming model for Hybrid Workflows: Combining task-based workflows and dataflows all-in-one

Ramon-Cortes, Cristian; Lordan, Francesc; Ejarque, Jorge; Badía, Rosa M.

doi:10.1016/j.future.2020.07.007

Cited by 12 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, various constraints in business processes and workflows often depend on the correctness of data. If the control flow is correct, then the data flow may not be correct; therefore, it is very important to analyze and restore the data flow in business processes and workflows [6]. Moreover, there is less research on data flow recovery.…”

Section: Related Workmentioning

confidence: 99%

“…Liu et al [12] have proposed a Petri net-based approach to model and analyze data flows. Ramon-Cortes et al [6] build a Distributed Stream Library supporting the integration of workflow and data flow to meet the needs of new Data Science workflows. Xiang et al [13] have proposed a PN-DOS model to reduce the accessibility of graphs to quickly detect data flow errors and ensure the correctness of business processes.…”

Section: Application Of Data Flowmentioning

confidence: 99%

See 1 more Smart Citation

Recovering Latent Data Flow from Business Process Model Automatically

Wang

Ali

et al. 2022

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

Process-driven applications evolve rapidly through the interaction between executable BPMN (Business Process Modeling and Notation) models, business tasks, and external services. Given these components operate on some shared process data, it is imperative to recover the latent data by visiting relation, which is known as data flow among these tasks. Data flow will benefit some typical applications including data flow anomaly checking and data privacy protection. However, in most cases, the complete data flow in a business process is not explicitly defined but hidden in model elements such as form declarations, variable declarations, and program code. Some methods to recovering data flow based on process model analysis of source code have some drawbacks; i.e., for security reasons, users do not want to provide source code but only encapsulated methods; therefore, data flows are difficult to analyze. We propose a method to generate running logs that are used to produce a complete data flow picture combined with the static code analysis method. This method combines the simple and easy-to-use characteristics of static code analysis methods and makes up for the shortcomings of static code analysis methods that cannot adapt to complex business processes, and as a result, the analyzed data flow is inaccurate. Moreover, a holistic framework is proposed to generate the data flow graph. The prototype system designed on Camunda and Flowable BPM (business process management) engine proves the applicability of the solution. The effectiveness of our method is validated on the prototype system.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Application Of Data Flowmentioning

confidence: 99%

Recovering Latent Data Flow from Business Process Model Automatically

Wang

Ali

et al. 2022

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

show abstract

“…However, developers frequently use the stream-oriented low-level frameworks, such as Apache Kafka [25], or dataflow models like Apache Storm [32], Apache Spark-Streaming [36] or Heron [26]. Apache Beam [4], COMPSs [30] and Twister2 [24] have gone a step further aiming to merge both workflows and dataflows in one single solution.…”

Section: Related Workmentioning

confidence: 99%

“…Converting the complex logic of the computation into an hybrid workflow -supporting both atomic and continuous-processing tasks -allows to parallelize and distribute the workload across the whole platform. For testing purposes and without loss of generality, our prototype leverages on COMPSs [27,30] to make this conversion, and we modified the COMPSs runtime to delegate the execution of the nested tasks onto Colony. Nevertheless, other programming models following a task-based approach, such as Swift [34] or Kepler [15], could also be integrated in the framework with the appropriate glue software.…”

Section: Introductionmentioning

confidence: 99%

Colony: Parallel Functions as a Service on the Cloud-Edge Continuum

Lordan

Lezzi

Badía

2021

Euro-Par 2021: Parallel Processing

Self Cite

View full text Add to dashboard Cite

Although smart devices markets are increasing their sales figures; their computing capabilities are not sufficient to provide goodenough-quality services. This paper proposes a solution to organize the devices within the Cloud-Edge Continuum in such a way that each one, as an autonomous individual -Agent-, processes events/data on its embedded compute resources while offering its computing capacity to the rest of the infrastructure in a Function-as-a-Service manner. Unlike other FaaS solutions, the described approach proposes to transparently convert the logic of such functions into task-based workflows backing on task-based programming models; thus, agents hosting the execution of the method generate the corresponding workflow and offloading part of the workload onto other agents to improve the overall service performance. On our prototype, the function-to-workflow transformation is performed by COMPSs; thus, developers can efficiently code applications of any of the three envisaged computing scenarios -sense-process-actuate, streaming and batch processing -throughout the whole Cloud-Edge Continuum without struggling with different frameworks specifically designed for each of them.

show abstract

“…Extensions of task-based programming to distributed programming, such as PyCOMPSs [23], [24], Dask [25], Ray [26], Parsl [27], and Pygion [28] are gaining popularity for scientific data analysis for the mix of performance and simplicity they offer. They provide a Python interface and often the transparent parallelization of some classical APIs (or part of them) like Numpy or Pandas.…”

Section: Related Workmentioning

confidence: 99%

DEISA: Dask-Enabled In Situ Analytics

Gueroudji

Raffin

2021

2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)

View full text Add to dashboard Cite

A widening performance gap is separating CPU performance and IO bandwidth on large scale systems. In some fields such as weather forecast and nuclear fusion, numerical models generate such amounts of data that classical post hoc processing is not feasible anymore due to the limits in both storage capacity and IO performance. In situ approaches are attractive to bypass disk accesses in these cases and fully leverage the HPC platform. They are however often complex to set up and can require to re-develop parallel versions of the analysis from scratch.In this paper we propose a hybrid model that is well suited for in situ workflows that combine regular simulations and irregular analytics. Our model couples the bulk synchronous parallel paradigm for simulation with a distributed task-based one for analysis. This reduces complexity and leverages the best of each of these two powerful paradigms. We validate the model with a prototype, called DEISA, that supports coupling MPI parallel codes with analyses written using Dask. This implementation requires minimal modifications of both the simulation and analysis codes compared to their post hoc counterpart. It give access to an already existing rich ecosystem to be used in situ such as the parallel versions of Numpy, Pandas and scikit-learn.Experiments in configurations up to 1024 cores show that DEISA can improve the simulation wallclock time (excluding analysis) by a factor up to 3 and the total experiment (including analysis) hour.core cost by a factor of up to 5 compared to parallel post hoc with plain Dask while requiring the modification of only two lines of python code, three of YAML, and none at all in a C simulation code already instrumented with PDI Data Interface.

show abstract

A programming model for Hybrid Workflows: Combining task-based workflows and dataflows all-in-one

Cited by 12 publications

References 13 publications

Recovering Latent Data Flow from Business Process Model Automatically

Recovering Latent Data Flow from Business Process Model Automatically

Colony: Parallel Functions as a Service on the Cloud-Edge Continuum

DEISA: Dask-Enabled In Situ Analytics

Contact Info

Product

Resources

About