Arteria: An automation system for a sequencing core facility

Dahlberg, Johan; Hermansson, Johan; Sturlaugsson, Steinar; Larsson, Pontus

doi:10.1101/214858

Cited by 3 publications

(3 citation statements)

References 13 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The usefulness of such an approach where an overarching framework provides primarily an orchestration role while calling out to other systems for the actual workflows is demonstrated by the Arteria project [18]. Arteria builds on the event-based StackStorm framework to allow triggering of external workflows based on any type of event, providing a flexible automation framework for sequencing core facilities.…”

Section: Resultsmentioning

confidence: 99%

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

et al. 2019

View full text Add to dashboard Cite

Background The complex nature of biological data has driven the development of specialized software tools. Scientific workflow management systems simplify the assembly of such tools into pipelines, assist with job automation, and aid reproducibility of analyses. Many contemporary workflow tools are specialized or not designed for highly complex workflows, such as with nested loops, dynamic scheduling, and parametrization, which is common in, e.g., machine learning. Findings SciPipe is a workflow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics, and other fields. SciPipe helps in particular with workflow constructs common in machine learning, such as extensive branching, parameter sweeps, and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on flow-based programming principles to support agile development of workflows based on a library of self-contained, reusable components. It supports running subsets of workflows for improved iterative development and provides a data-centric audit logging feature that saves a full audit trace for every output file of a workflow, which can be converted to other formats such as HTML, TeX, and PDF on demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline. Conclusions SciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine learning, through a flexible application programming interface suitable for scientists used to programming or scripting.

show abstract

Section: Resultsmentioning

confidence: 99%

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The usefulness of such an approach where an over-arching frameworks provide primarily an orchestration role while calling out to other systems for the actual workflows, is demonstrated by the Arteria project [18]. Arteria builds on the event-based StackStorm framework to allow triggering of external workflows based on any type of event, providing a flexible automation framework for sequencing core facilities.…”

Section: Resultsmentioning

confidence: 99%

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

Lampa

Dahlö

Alvarsson

et al. 2018

Preprint

View full text Add to dashboard Cite

Background: The complex nature of biological data has driven the development of specialized sof ware tools. Scienti c work ow management systems simplify the assembly of such tools into pipelines, assist with job automation and aid reproducibility of analyses. Many contemporary work ow tools are specialized and not designed for highly complex work ows, such as with nested loops, dynamic scheduling and parametrization, which is common in e.g. machine learning.Findings: SciPipe is a work ow programming library implemented in the programming language Go, for managing complex and dynamic pipelines in bioinformatics, cheminformatics and other elds. SciPipe helps in particular with work ow constructs common in machine learning, such as extensive branching, parameter sweeps and dynamic scheduling and parametrization of downstream tasks. SciPipe builds on Flow-based programming principles to support agile development of work ows based on a library of self-contained, re-usable components. It supports running subsets of work ows for improved iterative development, and provides a data-centric audit logging feature that saves a full audit trace for every output le of a work ow, which can be converted to other formats such as HTML, TeX and PDF on-demand. The utility of SciPipe is demonstrated with a machine learning pipeline, a genomics, and a transcriptomics pipeline.Conclusions: SciPipe provides a solution for agile development of complex and dynamic pipelines, especially in machine leaning, through a exible programming API suitable for scientists used to programming or scripting.

show abstract

“…The data set supporting the results of this article, “Reduced size Illumina NovaSeq runfolder,” is available in the Zenodo repository [23]. A snapshot of the code is also available via the GigaScience GigaDB repository [27].…”

Section: Availability Of Supporting Data and Materialsmentioning

confidence: 99%

Arteria: An automation system for a sequencing core facility

et al. 2019

Self Cite

View full text Add to dashboard Cite

BackgroundIn recent years, nucleotide sequencing has become increasingly instrumental in both research and clinical settings. This has led to an explosive growth in sequencing data produced worldwide. As the amount of data increases, so does the need for automated solutions for data processing and analysis. The concept of workflows has gained favour in the bioinformatics community, but there is little in the scientific literature describing end-to-end automation systems. Arteria is an automation system that aims at providing a solution to the data-related operational challenges that face sequencing core facilities.FindingsArteria is built on existing open source technologies, with a modular design allowing for a community-driven effort to create plug-and-play micro-services. In this article we describe the system, elaborate on the underlying conceptual framework, and present an example implementation. Arteria can be reduced to 3 conceptual levels: orchestration (using an event-based model of automation), process (the steps involved in processing sequencing data, modelled as workflows), and execution (using a series of RESTful micro-services). This creates a system that is both flexible and scalable. Arteria-based systems have been successfully deployed at 3 sequencing core facilities. The Arteria Project code, written largely in Python, is available as open source software, and more information can be found at https://arteria-project.github.io/ .ConclusionsWe describe the Arteria system and the underlying conceptual framework, demonstrating how this model can be used to automate data handling and analysis in the context of a sequencing core facility.

show abstract

Arteria: An automation system for a sequencing core facility

Cited by 3 publications

References 13 publications

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

SciPipe - A workflow library for agile development of complex and dynamic bioinformatics pipelines

Arteria: An automation system for a sequencing core facility

Contact Info

Product

Resources

About