Estimation of execution time for computing tasks

Bielecki, J.; Śmiałek, Michał

doi:10.1007/s10586-022-03774-1

Cited by 7 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the rapidly growing popularity of the Nextflow workflow management system, it is important to implement the available tools in the most effective way to maximize profit in execution time and computer resources ( 11 ). Published reports of genomic workflow comparisons are scarce and do not formally compare the same pipeline implemented with and without a workflow management system.…”

Section: Discussionmentioning

confidence: 99%

Nextflow vs. plain bash: different approaches to the parallelization of SNP calling from the whole genome sequence data

Sztuka,

Kotlarz,

Mielczarek

et al. 2024

NAR Genomics and Bioinformatics

View full text Add to dashboard Cite

This study compared computational approaches to parallelization of an SNP calling workflow. The data comprised DNA from five Holstein-Friesian cows sequenced with the Illumina platform. The pipeline consisted of quality control, alignment to the reference genome, post-alignment, and SNP calling. Three approaches to parallelization were compared: (i) a plain Bash script in which a pipeline for each cow was executed as separate processes invoked at the same time, (ii) a Bash script wrapped in a single Nextflow process and (iii) a Nextflow script with each component of the pipeline defined as a separate process. The results demonstrated that on average, the multi-process Nextflow script performed 15–27% faster depending on the number of assigned threads, with the biggest execution time advantage over the plain Bash approach observed with 10 threads. In terms of RAM usage, the most substantial variation was observed for the multi-process Nextflow, for which it increased with the number of assigned threads, while RAM consumption of the other setups did not depend much on the number of threads assigned for computations. Due to intermediate and log files generated, disk usage was markedly higher for the multi-process Nextflow than for the plain Bash and for the single-process Nextflow.

show abstract

Section: Discussionmentioning

confidence: 99%

Nextflow vs. plain bash: different approaches to the parallelization of SNP calling from the whole genome sequence data

Sztuka,

Kotlarz,

Mielczarek

et al. 2024

NAR Genomics and Bioinformatics

View full text Add to dashboard Cite

show abstract

“…With the rapidly growing popularity of the Nextflow workflow management system, it is important to implement the available tools in the most effective way to maximise profit in execution time and computer resources (Bielecki and Śmiałek 2023). Published reports of genomic workflow comparisons are scarce and do not formally compare the same pipeline implemented with and without a workflow management system.…”

Section: Discussionmentioning

confidence: 99%

Nextflow vs. plain Bash: Different Approaches to the Parallelisation of SNP Calling from the Whole Genome Sequence Data

Sztuka,

Kotlarz,

Mielczarek

et al. 2024

Preprint

View full text Add to dashboard Cite

This study compared computational approaches to parallelisation of an SNP calling workflow. Data comprised DNA from five Holstein-Friesian cows sequenced with the Illumina platform. The pipeline consisted of quality control, alignment to the reference genome, post-alignment, and SNP calling. Three approaches to parallelisation were compared: (i) a plain Bash script in which a pipeline for each cow was executed as separate processes invoked at the same time, (ii) a Bash script wrapped in a single Nextflow process, and (iii) a Nextflow script with each component of the pipeline defined as a separate process. The results demonstrated that on average, the multi-process Nextflow script performed 15% to 27% faster depending on the number of assigned threads, with the biggest execution time advantage over the plain Bash approach observed with 10 threads. In terms of RAM usage, the most substantial variation was observed for the multi-process Nextflow, for which it increased with the number of assigned threads, while RAM consumption of the other setups did not depend much on the numbers of threads assigned for computations. Due to intermediate and log files generated, disk usage was markedly higher for the multi-process Nextflow than for the plain Bash and for the single-process Nextflow.

show abstract

“…predicting computing time of dataset with different depth of coverage, read length and assembly complexity. 79 , 86 , 87 While acknowledging this consideration, it is important to note that conducting a benchmark on real machines provides practical insights and actionable data on a specific use case, despite potential confounders introduced by the machine itself or from software virtualization.…”

Section: Discussionmentioning

confidence: 99%

Easing genomic surveillance: A comprehensive performance evaluation of long-read assemblers across multi-strain mixture data of HIV-1 and Other pathogenic viruses for constructing a user-friendly bioinformatic pipeline

Wattanasombat,

Tongjai

2024

F1000Res

View full text Add to dashboard Cite

Background Determining the appropriate computational requirements and software performance is essential for efficient genomic surveillance. The lack of standardized benchmarking complicates software selection, especially with limited resources. Methods We developed a containerized benchmarking pipeline to evaluate seven long-read assemblers—Canu, GoldRush, MetaFlye, Strainline, HaploDMF, iGDA, and RVHaplo—for viral haplotype reconstruction, using both simulated and experimental Oxford Nanopore sequencing data of HIV-1 and other viruses. Benchmarking was conducted on three computational systems to assess each assembler’s performance, utilizing QUAST and BLASTN for quality assessment. Results Our findings show that assembler choice significantly impacts assembly time, with CPU and memory usage having minimal effect. Assembler selection also influences the size of the contigs, with a minimum read length of 2,000 nucleotides required for quality assembly. A 4,000-nucleotide read length improves quality further. Canu was efficient among de novo assemblers but not suitable for multi-strain mixtures, while GoldRush produced only consensus assemblies. Strainline and MetaFlye were suitable for metagenomic sequencing data, with Strainline requiring high memory and MetaFlye operable on low-specification machines. Among reference-based assemblers, iGDA had high error rates, RVHaplo showed the best runtime and accuracy but became ineffective with similar sequences, and HaploDMF, utilizing machine learning, had fewer errors with a slightly longer runtime. Conclusions The HIV-64148 pipeline, containerized using Docker, facilitates easy deployment and offers flexibility to select from a range of assemblers to match computational systems or study requirements. This tool aids in genome assembly and provides valuable information on HIV-1 sequences, enhancing viral evolution monitoring and understanding.

show abstract

Estimation of execution time for computing tasks

Cited by 7 publications

References 22 publications

Nextflow vs. plain bash: different approaches to the parallelization of SNP calling from the whole genome sequence data

Nextflow vs. plain bash: different approaches to the parallelization of SNP calling from the whole genome sequence data

Nextflow vs. plain Bash: Different Approaches to the Parallelisation of SNP Calling from the Whole Genome Sequence Data

Easing genomic surveillance: A comprehensive performance evaluation of long-read assemblers across multi-strain mixture data of HIV-1 and Other pathogenic viruses for constructing a user-friendly bioinformatic pipeline

Contact Info

Product

Resources

About