A Serverless Tool for Platform Agnostic Computational Experiment Management

Kiar, Gregory; Brown, Shawn T.; Glatard, Tristan; Evans, Alan C.

doi:10.3389/fninf.2019.00012

Cited by 18 publications

(14 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The emphasis on repeatability and reproducibility by building containerized solutions in clinical settings are starting to emerge in other fields, such as medical imaging and neurosciences [ 20 , 21 ]. Similarly, the benchmarking workflow is able to achieve high precision for both repeatability and reproducibility by being agnostic to the hardware infrastructure used to execute it.…”

Section: Discussionmentioning

confidence: 99%

Benchmarking workflows to assess performance and suitability of germline variant calling pipelines in clinical diagnostic assays

Krishnan

Utiramerur

et al. 2021

BMC Bioinformatics

View full text Add to dashboard Cite

Background Benchmarking the performance of complex analytical pipelines is an essential part of developing Lab Developed Tests (LDT). Reference samples and benchmark calls published by Genome in a Bottle (GIAB) consortium have enabled the evaluation of analytical methods. The performance of such methods is not uniform across the different genomic regions of interest and variant types. Several benchmarking methods such as hap.py, vcfeval, and vcflib are available to assess the analytical performance characteristics of variant calling algorithms. However, assessing the performance characteristics of an overall LDT assay still requires stringing together several such methods and experienced bioinformaticians to interpret the results. In addition, these methods are dependent on the hardware, operating system and other software libraries, making it impossible to reliably repeat the analytical assessment, when any of the underlying dependencies change in the assay. Here we present a scalable and reproducible, cloud-based benchmarking workflow that is independent of the laboratory and the technician executing the workflow, or the underlying compute hardware used to rapidly and continually assess the performance of LDT assays, across their regions of interest and reportable range, using a broad set of benchmarking samples. Results The benchmarking workflow was used to evaluate the performance characteristics for secondary analysis pipelines commonly used by Clinical Genomics laboratories in their LDT assays such as the GATK HaplotypeCaller v3.7 and the SpeedSeq workflow based on FreeBayes v0.9.10. Five reference sample truth sets generated by Genome in a Bottle (GIAB) consortium, six samples from the Personal Genome Project (PGP) and several samples with validated clinically relevant variants from the Centers for Disease Control were used in this work. The performance characteristics were evaluated and compared for multiple reportable ranges, such as whole exome and the clinical exome. Conclusions We have implemented a benchmarking workflow for clinical diagnostic laboratories that generates metrics such as specificity, precision and sensitivity for germline SNPs and InDels within a reportable range using whole exome or genome sequencing data. Combining these benchmarking results with validation using known variants of clinical significance in publicly available cell lines, we were able to establish the performance of variant calling pipelines in a clinical setting.

show abstract

Section: Discussionmentioning

confidence: 99%

Benchmarking workflows to assess performance and suitability of germline variant calling pipelines in clinical diagnostic assays

Krishnan

Utiramerur

et al. 2021

BMC Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Software pipelines were encapsulated and run using Singularity ( Kurtzer et al, 2017 ) version 2.6.1. Tasks were submitted, monitored, and provenance captured using Clowdr ( Kiar et al, 2019 ) version 0.1.2-1. All codes for performing the experiments and creating associated figures are available on GitHub at https://github.com/gkiar/stability and https://github.com/gkiar/stability-mca , respectively.…”

Section: Methodsmentioning

confidence: 99%

Comparing perturbation models for evaluating stability of neuroimaging pipelines

Kiar

Castro

Rioux

et al. 2020

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

With an increase in awareness regarding a troubling lack of reproducibility in analytical software tools, the degree of validity in scientific derivatives and their downstream results has become unclear. The nature of reproducibility issues may vary across domains, tools, data sets, and computational infrastructures, but numerical instabilities are thought to be a core contributor. In neuroimaging, unexpected deviations have been observed when varying operating systems, software implementations, or adding negligible quantities of noise. In the field of numerical analysis, these issues have recently been explored through Monte Carlo Arithmetic, a method involving the instrumentation of floating-point operations with probabilistic noise injections at a target precision. Exploring multiple simulations in this context allows the characterization of the result space for a given tool or operation. In this article, we compare various perturbation models to introduce instabilities within a typical neuroimaging pipeline, including (i) targeted noise, (ii) Monte Carlo Arithmetic, and (iii) operating system variation, to identify the significance and quality of their impact on the resulting derivatives. We demonstrate that even low-order models in neuroimaging such as the structural connectome estimation pipeline evaluated here are sensitive to numerical instabilities, suggesting that stability is a relevant axis upon which tools are compared, alongside more traditional criteria such as biological feasibility, computational efficiency, or, when possible, accuracy. Heterogeneity was observed across participants which clearly illustrates a strong interaction between the tool and data set being processed, requiring that the stability of a given tool be evaluated with respect to a given cohort. We identify use cases for each perturbation method tested, including quality assurance, pipeline error detection, and local sensitivity analysis, and make recommendations for the evaluation of stability in a practical and analytically focused setting. Identifying how these relationships and recommendations scale to higher order computational tools, distinct data sets, and their implication on biological feasibility remain exciting avenues for future work.

show abstract

“…All software developed for processing or evaluation is publicly available on GitHub at https://github.com/ gkpapers/2020ImpactOfInstability. Experiments were launched using Boutiques 42 and Clowdr 43 in Compute Canada’s HPC cluster environment. MCA instrumentation was achieved through Verificarlo 9 available on Github at https://github.com/verificarlo/verificarlo.…”

Section: Methodsmentioning

confidence: 99%

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Kiar

Chatelain

Castro

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The analysis of brain-imaging data requires complex and often non-linear transformations to support findings on brain function or pathologies. And yet, recent work has shown that variability in the choices that one makes when analyzing data can lead to quantitatively and qualitatively different results, endangering the trust in conclusions. Even within a given method or analytical technique, numerical instabilities could compromise findings. We instrumented a structural-connectome estimation pipeline with Monte Carlo Arithmetic, a technique to introduce random noise in floating-point computations, and evaluated the stability of the derived connectomes, their features, and the impact on a downstream analysis. The stability of results was found to be highly dependent upon which features of the connectomes were evaluated, and ranged from perfectly stable (i.e. no observed variability across executions) to highly unstable (i.e. the results contained no trustworthy significant information). While the extreme range and variability in results presented here could severely hamper our understanding of brain organization in brain-imaging studies, it also leads to an increase in the reliability of datasets. This paper highlights the potential of leveraging the induced variance in estimates of brain connectivity to reduce the bias in networks alongside increasing the robustness of their applications in the detection or classification of individual differences. This paper demonstrates that stability evaluations are necessary for understanding error and bias inherent to scientific computing, and that they should be a component of typical analytical workflows.

show abstract

A Serverless Tool for Platform Agnostic Computational Experiment Management

Cited by 18 publications

References 45 publications

Benchmarking workflows to assess performance and suitability of germline variant calling pipelines in clinical diagnostic assays

Benchmarking workflows to assess performance and suitability of germline variant calling pipelines in clinical diagnostic assays

Comparing perturbation models for evaluating stability of neuroimaging pipelines

Numerical Uncertainty in Analytical Pipelines Lead to Impactful Variability in Brain Networks

Contact Info

Product

Resources

About