Evaluating Workflow Management Systems: A Bioinformatics Use Case

Larsonneur, Élise; Mercier, Jonathan; Wiart, Nicolas; Floch, Édith Le; Delhomme, Olivier; Meyer, Vincent

doi:10.1109/bibm.2018.8621141

Cited by 20 publications

(18 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Niffler enables the execution of ML pipelines as Docker [17] containers on DICOM images retrieved from both PACS and the textual metadata of the images. The pipelines are run either through WDL specifications with a workflow manager that we developed extending Cromwell [18] or by simple shell scripts. The Real-Time DICOM Listener receives images from the primary PACS continuously as a DICOM imaging stream.…”

Section: Methodsmentioning

confidence: 99%

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images

Kathiravelu

Sharma

et al. 2021

J Digit Imaging

View full text Add to dashboard Cite

Real-time execution of machine learning (ML) pipelines on radiology images is difficult due to limited computing resources in clinical environments, whereas running them in research clusters requires efficient data transfer capabilities. We developed Niffler, an open-source Digital Imaging and Communications in Medicine (DICOM) framework that enables ML and processing pipelines in research clusters by efficiently retrieving images from the hospitals’ PACS and extracting the metadata from the images. We deployed Niffler at our institution (Emory Healthcare, the largest healthcare network in the state of Georgia) and retrieved data from 715 scanners spanning 12 sites, up to 350 GB/day continuously in real-time as a DICOM data stream over the past 2 years. We also used Niffler to retrieve images bulk on-demand based on user-provided filters to facilitate several research projects. This paper presents the architecture and three such use cases of Niffler. First, we executed an IVC filter detection and segmentation pipeline on abdominal radiographs in real-time, which was able to classify 989 test images with an accuracy of 96.0%. Second, we applied the Niffler Metadata Extractor to understand the operational efficiency of individual MRI systems based on calculated metrics. We benchmarked the accuracy of the calculated exam time windows by comparing Niffler against the Clinical Data Warehouse (CDW). Niffler accurately identified the scanners’ examination timeframes and idling times, whereas CDW falsely depicted several exam overlaps due to human errors. Third, with metadata extracted from the images by Niffler, we identified scanners with misconfigured time and reconfigured five scanners. Our evaluations highlight how Niffler enables real-time ML and processing pipelines in a research cluster.

show abstract

Section: Methodsmentioning

confidence: 99%

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images

Kathiravelu

Sharma

et al. 2021

J Digit Imaging

View full text Add to dashboard Cite

show abstract

“…While Pegasus is a versatile WMS, other WMS such as Nextflow (Di Tommaso et al, 2017) and Snakemake (Koster & Rahmann, 2012) are more commonly used for bioinformatics applications. However, compared to the other WMS, Pegasus has the best overall performance for efficiently utilizing the computational resources (Larsonneur et al, 2018). Moreover, Pegasus WMS provides a unique robust support of multiple computational platforms, varying from publicly available clusters to distributed cloud and grid infrastructures (Mitchell et al, 2019).…”

Section: Overview Of the Population Structure And Ecology For Salmonella Enterica Campylobacter Jejuni And Staphylococcus Aureusmentioning

confidence: 99%

ProkEvo: an automated, reproducible, and scalable framework for high-throughput bacterial population genomics analyses

Pavlovikj

Gomes-Neto

Deogun

et al. 2021

PeerJ

View full text Add to dashboard Cite

Whole Genome Sequence (WGS) data from bacterial species is used for a variety of applications ranging from basic microbiological research, diagnostics, and epidemiological surveillance. The availability of WGS data from hundreds of thousands of individual isolates of individual microbial species poses a tremendous opportunity for discovery and hypothesis-generating research into ecology and evolution of these microorganisms. Flexibility, scalability, and user-friendliness of existing pipelines for population-scale inquiry, however, limit applications of systematic, population-scale approaches. Here, we present ProkEvo, an automated, scalable, reproducible, and open-source framework for bacterial population genomics analyses using WGS data. ProkEvo was specifically developed to achieve the following goals: (1) Automation and scaling of complex combinations of computational analyses for many thousands of bacterial genomes from inputs of raw Illumina paired-end sequence reads; (2) Use of workflow management systems (WMS) such as Pegasus WMS to ensure reproducibility, scalability, modularity, fault-tolerance, and robust file management throughout the process; (3) Use of high-performance and high-throughput computational platforms; (4) Generation of hierarchical-based population structure analysis based on combinations of multi-locus and Bayesian statistical approaches for classification for ecological and epidemiological inquiries; (5) Association of antimicrobial resistance (AMR) genes, putative virulence factors, and plasmids from curated databases with the hierarchically-related genotypic classifications; and (6) Production of pan-genome annotations and data compilation that can be utilized for downstream analysis such as identification of population-specific genomic signatures. The scalability of ProkEvo was measured with two datasets comprising significantly different numbers of input genomes (one with ~2,400 genomes, and the second with ~23,000 genomes). Depending on the dataset and the computational platform used, the running time of ProkEvo varied from ~3-26 days. ProkEvo can be used with virtually any bacterial species, and the Pegasus WMS uniquely facilitates addition or removal of programs from the workflow or modification of options within them. To demonstrate versatility of the ProkEvo platform, we performed a hierarchical-based population structure analyses from available genomes of three distinct pathogenic bacterial species as individual case studies. The specific case studies illustrate how hierarchical analyses of population structures, genotype frequencies, and distribution of specific gene functions can be integrated into an analysis. Collectively, our study shows that ProkEvo presents a practical viable option for scalable, automated analyses of bacterial populations with direct applications for basic microbiology research, clinical microbiological diagnostics, and epidemiological surveillance.

show abstract

“…Previous comparative manuscripts tend to be largely descriptive 2, 6-9, 89 . The closest to our work is Larsonneur et al 16 , where testing was limited to a single node in a cluster, and examined WfMSs needed 4-6 minutes to run a genetic linkage analysis on variants from whole genome sequencing data of a trio 90 . This is a realistic bioinformatics use-case in terms of analysis complexity, but not in terms of computational requirements.…”

Section: Other Comparative Evaluationsmentioning

confidence: 99%

Design considerations for workflow management systems use in production genomics research and the clinic

Ahmed

Allen

Bhat

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. Results: This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, "which WfMS should be chosen for a given bioinformatics application regardless of analysis type?". Conclusions: The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.

show abstract

Evaluating Workflow Management Systems: A Bioinformatics Use Case

Cited by 20 publications

References 7 publications

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images

ProkEvo: an automated, reproducible, and scalable framework for high-throughput bacterial population genomics analyses

Design considerations for workflow management systems use in production genomics research and the clinic

Contact Info

Product

Resources

About