Generating Complex Astronomy Workflows

Berriman, G. Bruce; Deelman, Ewa; Good, J. C.; Jacob, Joseph; Laity, A. C.; Prince, Thomas A.; Singh, Gurmeet; Su, Min

doi:10.1007/978-1-84628-757-2_3

Cited by 15 publications

(12 citation statements)

References 153 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the Amazon EC2 the Storm deployment used an 18-worker node setup. We chose Amazon's T2.medium instances 28 , provisioned with 2 vCPUs and 4GB of RAM. Amazon instances are built on Intel Xeon processors operating at 2.5GHz, with Turbo up to 3.3GHz.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

`dispel4py`: A Python framework for data-intensive scientific computing

Filguiera

Krause

Atkinson

et al. 2016

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and HPC architectures and consistent scalability.

show abstract

Section: Methodsmentioning

confidence: 99%

“…These systems have achieved substantial progress in handling data-intensive scientific computations; e.g. in astrophysics [26,27,28], in climate physics and meteorology [29], in biochemistry [31], in geosciences and ge-oengineering [32] and in environmental sciences [33,34].…”

Section: Introductionmentioning

confidence: 99%

`dispel4py`: A Python framework for data-intensive scientific computing

Filguiera

Krause

Atkinson

et al. 2016

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…Data-driven workflows have been used to guide data processing in a variety of fields. Some examples are astronomy [7], biology [8], clinical research [9], archive scanning [10], telecommunications [11], banking [12] and process industry [13], to name a few. The common methodology in all these fields consists of following a number of steps to prepare a dataset for data mining.…”

Section: Abstract-compositionmentioning

confidence: 99%

Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA

Salvador

Budka

Gabrys

2019

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

Composition and parametrisation of multicomponent predictive systems (MCPSs) consisting of chains of data transformation steps is a challenging task. Auto-WEKA is a tool to automate the Combined Algorithm Selection and Hyperparameter (CASH) optimisation problem. In this paper we extend the CASH problem and Auto-WEKA to support MCPS including preprocessing steps for both classification and regression tasks. We define the optimisation problem in which the search space consists of suitably parametrised Petri nets forming the sought MCPS solutions. In the experimental analysis we focus on examining the impact of considerably extending the search space (from approximately 22,000 to 812 billion possible combinations of methods and categorical hyperparameters). In a range of extensive experiments three different optimisation strategies are used to automatically compose MCPSs for 21 publicly available datasets. The diversity of the composed MCPSs found is an indication that fully and automatically exploiting different combinations of data cleaning and preprocessing techniques is possible and highly beneficial for different predictive models. We also present the results on 7 datasets from real chemical production processes. Our findings can have a major impact on development of high quality predictive models as well as their maintenance and scalability aspects needed in modern applications and deployment scenarios.

show abstract

“…Success stories from the scientific communities demonstrate the benefits of combining efforts in terms of technology and knowledge to solve grand challenges, e.g., in astronomy [7], physics [8], meteorology [32], etc. A wide range of workflow management systems (WMS) support these e-Science activities.…”

Section: Related Workmentioning

confidence: 99%

Towards optimising distributed data streaming graphs using parallel streams

Liew

Atkinson

Hemert

et al. 2010

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

View full text Add to dashboard Cite

Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multidisciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We identify performance metrics and design a measurement tool to evaluate each enactment. We conducted experiments to evaluate our optimisation strategies with a real world problem in the Life Sciences-EURExpress-II. The paper presents our distributed data-handling model, the optimisation and instrumentation strategies and the evaluation experiments. We demonstrate linear speed up and argue that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.

show abstract

Generating Complex Astronomy Workflows

Cited by 15 publications

References 153 publications

`dispel4py`: A Python framework for data-intensive scientific computing

`dispel4py`: A Python framework for data-intensive scientific computing

Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA

Towards optimising distributed data streaming graphs using parallel streams

Contact Info

Product

Resources

About

Generating Complex Astronomy Workflows

Cited by 15 publications

References 153 publications

dispel4py: A Python framework for data-intensive scientific computing

dispel4py: A Python framework for data-intensive scientific computing

Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA

Towards optimising distributed data streaming graphs using parallel streams

Contact Info

Product

Resources

About

`dispel4py`: A Python framework for data-intensive scientific computing

`dispel4py`: A Python framework for data-intensive scientific computing