Containers for Portable, Productive, and Performant Scientific Computing

Hale, Jack; Li, Lizao; Richardson, Chris N.; Wells, Garth N.

doi:10.1109/mcse.2017.2421459

“…“Shifter” (as of September 2016 only available as pre-release beta version) has been discussed in the context of running containerized academic software [51]. The principles behind Shifter are similar to Singularity, but Shifter also attempts to tackle the problem of managing the container images.…”

Section: Discussionmentioning

confidence: 99%

BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods

Gorgolewski

¹

,

Alfaro-Almagro

²

,

Auer

³

et al. 2017

View full text Add to dashboard Cite

The rate of progress in human neurosciences is limited by the inability to easily apply a wide range of analysis methods to the plethora of different datasets acquired in labs around the world. In this work, we introduce a framework for creating, testing, versioning and archiving portable applications for analyzing neuroimaging data organized and described in compliance with the Brain Imaging Data Structure (BIDS). The portability of these applications (BIDS Apps) is achieved by using container technologies that encapsulate all binary and other dependencies in one convenient package. BIDS Apps run on all three major operating systems with no need for complex setup and configuration and thanks to the comprehensiveness of the BIDS standard they require little manual user input. Previous containerized data processing solutions were limited to single user environments and not compatible with most multi-tenant High Performance Computing systems. BIDS Apps overcome this limitation by taking advantage of the Singularity container technology. As a proof of concept, this work is accompanied by 22 ready to use BIDS Apps, packaging a diverse set of commonly used neuroimaging algorithms.

show abstract

“…To facilitate reproducibility of data processing and the execution of pipelines in any computing environment, DolphinNext leverages Nextflow's support for Singularity and Docker container technologies (25,26). This allows the execution of a pipeline created by DolphinNext to require only Nextflow and a container software (Singularity or Docker) to be installed in the host machine.…”

Section: Methodsmentioning

confidence: 99%

“…To easily enable third parties to reproduce analyses, complete pipelines along with the computing environment used must be available and need to be easily shared (21)(22)(23)(24). DolphinNext allows users to package pipelines into portable containers (25,26) that can be run as stand-alone applications because they include the exact versions of all software dependencies that were tested and used. The automatic inclusion of all software dependencies vastly simplifies the effort needed to share, run and reproduce the exact results obtained by a pipeline.…”

Section: Introductionmentioning

confidence: 99%

DolphinNext: A graphical user interface for creating, deploying and executing Nextflow pipelines

Yukselen

¹

,

Turkyilmaz

²

,

Öztürk

³

et al. 2019

Preprint

View full text Add to dashboard Cite

Several platforms currently exist for the design and execution of complex pipelines (e.g. Galaxy, GenePattern, GeneProf). Unfortunately, these platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, Nextflow was implemented to simplify portable, scalable, and reproducible scientific pipelines using containers. We have used Nextflow capabilities as leverage and developed a user interface, DolphinNext, for creating, deploying, and executing complex Nextflow pipelines for high throughput data processing. The guiding principle of DolphinNext is to facilitate the building and deployment of complex pipelines using a modular approach implemented in a graphical interface. DolphinNext provides: 1. A drag and drop user interface that abstracts pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. A user interface to monitor pipeline execution that allows the re-initiation of pipelines at intermediate steps 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Seamless portability to distributed computational environments such as high-performance clusters or cloud computing environments.

show abstract

“…Singularity is not the only solution proposed to handle containers on HPCs. "Shifter" (as of September 2016 only available as pre-release beta version) has been discussed in the context of running containerized academic software (Hale et al 2016) . The principles behind Shifter are similar to Singularity, but Shifter also attempts to tackle the problem of managing the container images.…”

Section: App Name Description Applicable Modalitiesmentioning

confidence: 99%

BIDS Apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods

Gorgolewski

¹

,

Alfaro-Almagro

²

,

Auer

³

et al. 2016

Preprint

View full text Add to dashboard Cite

The rate of progress in human neurosciences is limited by the inability to easily apply a wide range of analysis methods to the plethora of different datasets acquired in labs around the world. In this work, we introduce a framework for creating, testing, versioning and archiving portable applications for analyzing neuroimaging data organized and described in compliance with the Brain Imaging Data Structure (BIDS). (BIDS Apps) is achieved by using container technologies that encapsulate all binary and other dependencies in one convenient package. BIDS Apps run on all three major operating systems with no need for complex setup and configuration and thanks to the comprehensiveness of the BIDS standard they require little manual user input. Previous containerized data processing solutions were limited to single user environments and not compatible with most multi-tenant High Performance Computing systems. BIDS Apps overcome this limitation by taking advantage of the Singularity container technology. As a proof of concept, this work is accompanied by 22 ready to use BIDS Apps, packaging a diverse set of commonly used neuroimaging algorithms. Author summaryMagnetic Resonance Imaging (MRI) is a non-invasive way to measure human brain structure and activity that has been used for over 25 years. There are thousands MRI studies performed every year generating a substantial amount of data. At the same time, many new data analysis methods are being developed every year. The potential of using new analysis methods on the variety of existing and newly acquired data is hindered by difficulties in software deployment and lack of support for standardized input data. Here we propose to use container technology to make deployment of a wide range of data analysis techniques easy. In addition, we adapt the existing data analysis tools to interface with data organized in a standardized way. We hope that this approach will enable researchers to access a wider range of methods when analyzing their data which will lead to accelerated progress in human neuroscience.

show abstract

Containers for Portable, Productive, and Performant Scientific Computing

Cited by 47 publications

References 12 publications

BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods

BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods

DolphinNext: A graphical user interface for creating, deploying and executing Nextflow pipelines

BIDS Apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods

Contact Info

Product

Resources

About