Interoperable and scalable data analysis with microservices: Applications in Metabolomics

Khoonsari, Payam Emami; Moreno, Pablo; Bergmann, Sven; Burman, Joachim; Capuccini, Marco; Carone, Matteo; Cascante, Marta; Atauri, Pedro de; Dudová, Zdenka; Foguet, Carles; González-Beltrán, Alejandra; Hankemeier, Thomas; Haug, Kenneth; He, Sijin; Herman, Stephanie; Johnson, David; Kale, Namrata; Larsson, Anders; Salek, Reza M.; Neumann, Steffen; Peters, Kristian; Pireddu, Luca; Rocca-Serra, Philippe; Roger, Pierrick; Rueedi, Rico; Ruttkies, Christoph; Sadawi, Noureddin; Sansone, Susanna-Assunta; Schober, Daniel; Selivanov, Vitaly A.; Thévenot, Etienne; Vliet, Michael Van; Zanetti, Gianluigi; Steinbeck, Christoph; Kultima, Kim; Spjuth, Ola

doi:10.1101/213603

Cited by 3 publications

(2 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PhenoMeNal has been able to adapt Galaxy for use with a microservices-based architecture [31]. To this end, modules are encapsulated into Docker containers that can be flexibly launched within the cloud e-infrastructure.…”

Section: Methodsmentioning

confidence: 99%

“…In PhenoMeNal, we have extended Galaxy, Jupyter, Luigi, and Pachyderm in such a way that they can be orchestrated throughout the cloud infrastructure together with the data analysis tools themselves [31]. Six important metabolomics workflows have been fully integrated into PhenoMeNal (Table 1), and more (mzQuality, NMR-BATMAN) are available for testing.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

PhenoMeNal: processing and analysis of metabolomics data in the cloud

et al. 2018

Self Cite

View full text Add to dashboard Cite

BackgroundMetabolomics is the comprehensive study of a multitude of small molecules to gain insight into an organism's metabolism. The research field is dynamic and expanding with applications across biomedical, biotechnological, and many other applied biological domains. Its computationally intensive nature has driven requirements for open data formats, data repositories, and data analysis tools. However, the rapid progress has resulted in a mosaic of independent, and sometimes incompatible, analysis methods that are difficult to connect into a useful and complete data analysis solution.FindingsPhenoMeNal (Phenome and Metabolome aNalysis) is an advanced and complete solution to set up Infrastructure-as-a-Service (IaaS) that brings workflow-oriented, interoperable metabolomics data analysis platforms into the cloud. PhenoMeNal seamlessly integrates a wide array of existing open-source tools that are tested and packaged as Docker containers through the project's continuous integration process and deployed based on a kubernetes orchestration framework. It also provides a number of standardized, automated, and published analysis workflows in the user interfaces Galaxy, Jupyter, Luigi, and Pachyderm.ConclusionsPhenoMeNal constitutes a keystone solution in cloud e-infrastructures available for metabolomics. PhenoMeNal is a unique and complete solution for setting up cloud e-infrastructures through easy-to-use web interfaces that can be scaled to any custom public and private cloud environment. By harmonizing and automating software installation and configuration and through ready-to-use scientific workflow user interfaces, PhenoMeNal has succeeded in providing scientists with workflow-driven, reproducible, and shareable metabolomics data analysis platforms that are interfaced through standard data formats, representative datasets, versioned, and have been tested for reproducibility and interoperability. The elastic implementation of PhenoMeNal further allows easy adaptation of the infrastructure to other application areas and ‘omics research domains.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

PhenoMeNal: processing and analysis of metabolomics data in the cloud

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

Container-based bioinformatics with Pachyderm

Novella

Khoonsari

Herman

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Motivation:Computational biologists face many challenges related to data size, and they need to manage complicated analyses often including multiple stages and multiple tools, all of which must be deployed to modern infrastructures. To address these challenges and maintain reproducibility of results, researchers need (i) a reliable way to run processing stages in any computational environment, (ii) a well-defined way to orchestrate those processing stages, and (iii) a data management layer that tracks data as it moves through the processing pipeline. Results:Pachyderm is an open-source workflow system and data management framework that fulfills these needs by creating a data pipelining and data versioning layer on top of projects from the container ecosystem, having Kubernetes as the backbone for container orchestration. We adapted Pachyderm and demonstrated its attractive properties in bioinformatics. A Helm Chart was created so that researchers can use Pachyderm in multiple scenarios. The Pachyderm File System was extended to support block storage. A wrapper for initiating Pachyderm on cloud-agnostic virtual infrastructures was created.

show abstract

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Peters

Bradbury

Bergmann

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Metabolomics is the comprehensive study of a multitude of small molecules to gain insight into an organism's metabolism. The research field is dynamic and expanding with applications across biomedical, biotechnological and many other applied biological domains. Its computationally-intensive nature has driven requirements for open data formats, data repositories and data analysis tools. However, the rapid progress has resulted in a mosaic of independent -and sometimes incompatible -analysis methods that are difficult to connect into a useful and complete data analysis solution. Findings: The PhenoMeNal (Phenome and Metabolome aNalysis) e-infrastructure provides a complete, workflow-oriented, interoperable metabolomics data analysis solution for a modern infrastructure-as-a-service (IaaS) cloud platform. PhenoMeNal seamlessly integrates a wide array of existing open source tools which are tested and packaged as Docker containers through the project's continuous integration process and deployed based on a kubernetes orchestration framework. It also provides a number of standardized, automated and published analysis workflows in the user interfaces Galaxy, Jupyter, Luigi and Pachyderm. Conclusions: PhenoMeNal constitutes a keystone solution in cloud infrastructures available for metabolomics. It provides scientists with a ready-to-use, workflow-driven, reproducible and shareable data analysis platform harmonizing the software installation and configuration through user-friendly web interfaces. The deployed cloud environments can be dynamically scaled to enable large-scale analyses which are interfaced through standard data formats, versioned, and have been tested for reproducibility and interoperability. The flexible implementation of PhenoMeNal allows easy adaptation of the infrastructure to other application areas and 'omics research domains.

show abstract

Interoperable and scalable data analysis with microservices: Applications in Metabolomics

Cited by 3 publications

References 53 publications

PhenoMeNal: processing and analysis of metabolomics data in the cloud

PhenoMeNal: processing and analysis of metabolomics data in the cloud

Container-based bioinformatics with Pachyderm

PhenoMeNal: Processing and analysis of Metabolomics data in the Cloud

Contact Info

Product

Resources

About