Towards Continuous Workflow Enactment Systems

Neophytou, Panayiotis; Chrysanthis, Panos K.; Labrinidis, Alexandros

doi:10.1007/978-3-642-03354-4_13

Cited by 5 publications

(11 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But this capability is not adequate for updating the logic of tasks and the dataflow's structure at runtime. Stream processing systems [23], [24], [25] complement workflow systems by operating on realtime data. Online update problems in the context of streaming systems are as less studied.…”

Section: Implementation and Resultsmentioning

confidence: 99%

Continuous Dataflow Update Strategies for Mission-Critical Applications

Wickramaarachchi

Simmhan

2013

2013 IEEE 9th International Conference on E-Science

View full text Add to dashboard Cite

Continuous dataflows complement scientific workflows by allowing composition of realtime data ingest and analytics pipelines to process data streams from pervasive sensors and "always-on" scientific instruments. Such dataflows are missioncritical applications that cannot suffer downtime, need to operate consistently, and are long running, but may need to be updated to fix bugs or add features. This poses the problem: How do we update the continuous dataflow application with minimal disruption? In this paper, we formalize different types of dataflow update models for continuous dataflow applications, and identify the qualitative and quantitative metrics to be considered when choosing an update strategy. We propose five dataflow update strategies, and analytically characterize their performance tradeoffs. We validate one of these consistent, low-latency update strategies using the F oε dataflow engine for an eEngineering application from the Smart Power Grid domain, and show its relative performance benefits against a naïve update strategy.

show abstract

Section: Implementation and Resultsmentioning

confidence: 99%

Continuous Dataflow Update Strategies for Mission-Critical Applications

Wickramaarachchi

Simmhan

2013

2013 IEEE 9th International Conference on E-Science

View full text Add to dashboard Cite

show abstract

“…This ability allows operators such as aggregation and difference to be used in recursive loops, which are useful for expressing branch and bound execution strategies. Our progress-detection mechanisms may also be of use in other settings with cyclic event processing, such as continuous workflow systems [28].…”

Section: Discussionmentioning

confidence: 99%

On-the-fly progress detection in iterative stream queries

2009

View full text Add to dashboard Cite

Multiple researchers have proposed cyclic query plans for evaluating iterative queries over streams or rapidly changing input. The Declarative Networking community uses cyclic plans to evaluate Datalog programs that track reachability and other graph traversals on networks. Cyclic query plans can also evaluate pattern-matching and other queries based on event sequences.An issue with cyclic queries over dynamic inputs is knowing when the query result has progressed to a certain point in the input, since the number of iterations is data dependent. One option is a -strictly staged‖ computation, where the query plan quiesces between inputs. This option introduces significant latency, and may also -underload‖ inter-operator buffers. An alternative is to settle for soft guarantees, such as -eventual consistency‖. Such imprecision can make it difficult, for example, to know when to purge state from stateful operators.We propose a third option in which cyclic queries run continuously, but detect progress -on the fly‖ by means of a Flying Fixed-Point (FFP) operator. FFP sits on the cyclic loop and circulates speculative predictions on forward progress, which it then validates. FFP is always able to track progress for a class of queries we term strongly convergent. A key advantage of FFP is that it works with existing algebra operators, thereby inheriting their capabilities, such as windowing and dealing with out-oforder input. Also, for stream systems that explicitly model inputevent lifetimes, we know exactly which values are in the query result at each point in time.A key implementation decision is the method for speculating. Using the high-water mark of data events minimizes the number of speculative punctuations. Probing operators on the cyclic loop to determine their external progress circulates many more speculative messages, but tracks actual output progress more closely. We show how a hybrid approach limits predictions while coming close the progress-tracking ability of Probing.

show abstract

“…Continuous workflows can potentially run for an unlimited amount of time, constantly monitoring data streams. To achieve that, our Continuous Workflow Model that we have proposed in [5], introduced:…”

Section: Continuous Worflow Modelmentioning

confidence: 99%

“…In our previous work in [5] we have examined the capability of current workflow models and workflow management systems to support business and scientific reactive applications. We based our examination on the Workflow Pattern framework [8].…”

Section: Introductionmentioning

confidence: 99%

“…The main difference between traditional and continuous workflows is that the latter are continuously (i.e., always) active and continuously reacting on internal streams of events and external streams of updates from multiple sources, at the same time and in any part of the workflow network. We have shown in [5] how the workflow and communications patterns are cast into continuous workflows (CWfs) and also proposed four new CWfs patterns.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CONFLuEnCE

Neophytou

Chrysanthis

Labrinidis

2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Traditional workflow enactment systems view a workflow as a onetime interaction with various data sources, executing a series of steps once, whenever the workflow results are requested. The fundamental underlying assumption has been that data sources are passive and all interactions are structured along the request/reply (query) model. Hence, traditional Workflow Management Systems cannot effectively support business or scientific reactive applications that require the processing of continuous data streams.In this demo, we will present our prototype which transforms workflow execution from the traditional step-wise workflow execution model to a continuous execution model, in order to handle data streams published and delivered asynchronously from multiple sources. We will demonstrate a supply chain management scenario which takes advantage of our continuous execution model to enable on-line interaction between different user roles as well as streaming data coming from various sources.

show abstract

Towards Continuous Workflow Enactment Systems

Cited by 5 publications

References 10 publications

Continuous Dataflow Update Strategies for Mission-Critical Applications

Continuous Dataflow Update Strategies for Mission-Critical Applications

On-the-fly progress detection in iterative stream queries

CONFLuEnCE

Contact Info

Product

Resources

About