Time-bound analytic tasks on large datasets through dynamic configuration of workflows

Abstract

Domain experts are often untrained in big data technologies and this limits their ability to exploit the data they have available. Workflow systems hide the complexities of high-end computing and software engineering by offering pre-packaged analytic steps combined into multi-step methods commonly used by experts. A current limitation of workflow systems is that they do not take into account user deadlines: they run workflows selected by the user, but take their time to do so. This is impractical when large datasets are at stake, since users often prefer to see an answer faster even if it has lower precision or quality. In this paper, we present an extension to workflow systems that enables them to take into account user deadlines by automatically generating alternative workflow candidates and ranking them according to performance estimates. The system makes these estimates based on workflow performance models created from workflow executions, and uses semantic technologies to reason about workflow options. Possible workflow candidates are presented to the user in a compact manner, and are ranked according to their runtime estimates. We have implemented this approach in the WOOT system, which combines and extends capabilities from the WINGS semantic workflow system and the Apache OODT Object Oriented Data Technology and workflow execution system.

A Taxonomic Analysis of Perspectives in Generating Space-Time Research Questions in Environmental Sciences

“…Enabling faster scientific investigations is a central theme guiding the e-Research initiatives (Hey & Trefethen, 2005). Domain scientists have been worked with computer scientists to build workflow systems for complex data analytics (Deelman, Gannon, Shields, & Taylor, 2009;Gil, Ratnakar, & Fritz, 2010). Our taxonomic analysis could provide guidance for building these analytic workflows.…”

Towards Automating Data Narratives

“…WINGS represents high-level abstract workflow templates and uses them to generate executable workflows [27]. Figure 3 illustrates the difference, using a topic training workflow where the most popular topic trends are extracted from a collection of documents [29]. Figure 3 shows an example of a workflow template at the bottom.…”