HyperLoom

Cima, Vojtech; Böhm, Stanislav; Martinovič, Jan; Dvorský, Jiří; Janurová, Kateřina; Aa, Tom Vander; Ashby, Thomas J.; Chupakhin, Vladimir

doi:10.1145/3183767.3183768

“…Second, with systems like Anduril 6 , Balsam 7 , Hyperloom 8 , Jug 9 , Pwrake 10 , Ruffus 11 , SciPipe 12 , SCOOP 13 , and COMPSs 14 workflows are specified using a set of classes and functions for generic programming languages like Python, Scala, and others. Such systems have the advantage that they can be used without a graphical interface (e.g.…”

Section: Introductionmentioning

confidence: 99%

Sustainable data analysis with Snakemake

Mölder

¹

,

Jablonski

²

,

Letcher

³

et al. 2021

View full text Add to dashboard Cite

Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.

show abstract

“…Second, with systems like Anduril 7 , Balsam 8 , Hyperloom 9 , Jug 10 , Pwrake 11 , Ruffus 12 , SciPipe 13 , SCOOP 14 , and COMPSs 15 , and JUDI 16 , workflows are specified using a set of classes and functions for generic programming languages like Python, Scala, and others. Such systems have the advantage that they can be used without a graphical interface (e.g.…”

Section: Introductionmentioning

confidence: 99%

Sustainable data analysis with Snakemake

Mölder

¹

,

Jablonski

²

,

Letcher

³

et al. 2021

View full text Add to dashboard Cite

Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.

show abstract

“…The aim of this article is to introduce an implementation of a distributed hyperparameter search for the vehicle routing problem solver based on HyperLoo m [8]. Using HyperLoo m p ipeline we are able to sweep through large parameter spaces and discover the optimal solution for the heuristic algorith m. HyperLoo m can discover the optimal solution for the heuristic algorithm.…”

Section: Introductionmentioning

confidence: 99%

Setting the Configuration Parameters of the Algorithm for the Periodic Vehicle Routing Problem by HPC Power

Grakova

¹

,

Martinovič

²

,

Slaninová

³

et al. 2019

MATEC Web Conf.

Self Cite

0

View full text Add to dashboard Cite

The quality of an opt imal solut ion of the Vehicle Rout ing Problem is st rongly depended on the sett ing of the configurat ion parameters of the algorithm. The paper is focused on the int roduct ion of hyperparameter search for solving the Vehicle Rout ing Problem using a HyperLoom plat form for defining and execut ing scient ific pipelines in a dist ributed environment . To give a concrete example, we focused on Periodic Vehicle Rout ing Problem for the waste collect ion. HyperLoom plat form was used to define and execute the hyperparameters sweep pipeline. The heurist ic algorithm was tested on a real benchmark of the waste collect ion in Ostrava, Czech Republic. The aim of our ca se was to effect ively combine the minimizat ion of the total t ravelled distance and the opt imizat ion of the fairness of the routes in terms of the standard deviat ion of a tour length. The waste collect ion problem was very extensive and computat ionally demanding, so it was necessary to use high performance comput ing architecture for test ing a large number of different sett ings of configurat ion parameters. The experiments were run on the supercomputer Salomon operated by IT4Innovations Nat ional Supercomput ing Center in the Czech Republic.

show abstract

HyperLoom

Cited by 13 publications

References 7 publications

Sustainable data analysis with Snakemake

Sustainable data analysis with Snakemake

Sustainable data analysis with Snakemake

Setting the Configuration Parameters of the Algorithm for the Periodic Vehicle Routing Problem by HPC Power

Contact Info

Product

Resources

About