Streamlining data-intensive biology with workflow systems

Abstract

As the scale of biological data generation has increased, the bottleneck of research has shifted from data generation to analysis. Researchers commonly need to build computational workflows that include multiple analytic tools and require incremental development as experimental insights demand tool and parameter modifications. These workflows can produce hundreds to thousands of intermediate files and results that must be integrated for biological insight. Data-centric workflow systems that internally manage computational resources, software, and conditional execution of analysis steps are reshaping the landscape of biological data analysis and empowering researchers to conduct reproducible analyses at scale. Adoption of these tools can facilitate and expedite robust data analysis, but knowledge of these techniques is still lacking. Here, we provide a series of strategies for leveraging workflow systems with structured project, data, and resource management to streamline large-scale biological analysis. We present these practices in the context of high-throughput sequencing data analysis, but the principles are broadly applicable to biologists working beyond this field.

Analyzing Modern Biomolecules: The Revolution of Nucleic-Acid Sequencing – Review

“…Indeed, there is currently a new revolution, as shown by the exponential growth of publications, regarding the possibility to sequence DNA and RNA from since-cells, as well as single organelles (mitochondria and chloroplasts) [13]. Special emphasis is now focused on integrating different -omics technologies, such as genomics (usually, DNA), transcriptomics (RNA), proteomics (peptides, like proteins), epigenomics (epigenetic factors) and metabolomics (metabolites), that eventually influence phenotypes in health and disease [14][15][16][17]. Furthermore, a combination of multi-omics techniques, complemented with morphological and physiological ones, allows a holistic approach to deciphering biological systems [18,19].…”

Tourmaline: a containerized workflow for rapid and iterable amplicon sequence analysis using QIIME 2 and Snakemake

“…The notebook generates Pandas DataFrames for the feature table, taxonomy, reference sequence properties, and metadata, and a Pandas Series for alpha diversity. Static plots are generated from some of these tables using Seaborn (Qalieh et al 2017).…”

“…A popular approach for standardizing amplicon data analysis is to develop analysis pipelines or workflows (Prodan et al 2020;Reiter et al 2021) to run on local or networked computing resources or in the cloud. Examples include Anacapa (Curd et al 2019), Banzai (https://github.com/jimmyodonnell/banzai), PEMA (Zafeiropoulos et al 2020), Ampliseq (Straub et al 2020), Cascabel (Asbun et al 2019), and dadasnake (Weißbecker, Schnabel and Heintz-Buschart 2020).…”

Ten simple rules to cultivate transdisciplinary collaboration in data science

“…Lessons learned were summarized and developed into the following interconnected (see Fig 1) Ten Simple Rules within a "collaboratory cultures" framework. While many Ten Simple Rules have been written about general collaboration, data sciences collaboration, statisticians' collaborations, and leveraging big data [2][3][4][5][6][7], we emphasize the "nontechnical" criteria that are necessary to promote effective collaborations, accelerate discovery, facilitate new partnerships, and develop the role of individuals within transdisciplinary [8] research projects-projects that combine disciplines in a nontraditional way, resulting in the development of novel frameworks, concepts, and methodologies to address scientific problems.…”