NeatFreq: reference-free data reduction and coverage normalization for De Novosequence assembly

McCorrison, Jamison; Venepally, Pratap; Singh, Indresh; Fouts, Derrick E.; Lasken, Roger S.; Methé, Barbara A.

doi:10.1186/s12859-014-0357-3

Cited by 18 publications

(15 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although coverage reduction has been primarily used for unbalanced data ( Brown et al 2012 ), we have shown in Lonardi et al (2015) that in the presence of ultra-deep sequencing data, the assembly of a random sample of the input reads only marginally improves the assembly quality compared with the assembly of entire dataset. Diginorm ( Brown et al 2012 ) and NeatFreq ( McCorrison et al 2014 ) are two examples of down-sampling methods aimed to produce a more uniform coverage. They both reduce coverage by selecting representative reads binned by their median k-mer frequency.…”

Section: Introductionmentioning

confidence: 99%

De novo meta-assembly of ultra-deep sequencing data

Mirebrahim

Lonardi

2015

Bioinformatics

View full text Add to dashboard Cite

We introduce a new divide and conquer approach to deal with the problem of de novo genome assembly in the presence of ultra-deep sequencing data (i.e. coverage of 1000x or higher). Our proposed meta-assembler Slicembler partitions the input data into optimal-sized ‘slices’ and uses a standard assembly tool (e.g. Velvet, SPAdes, IDBA_UD and Ray) to assemble each slice individually. Slicembler uses majority voting among the individual assemblies to identify long contigs that can be merged to the consensus assembly. To improve its efficiency, Slicembler uses a generalized suffix tree to identify these frequent contigs (or fraction thereof). Extensive experimental results on real ultra-deep sequencing data (8000x coverage) and simulated data show that Slicembler significantly improves the quality of the assembly compared with the performance of the base assembler. In fact, most of the times, Slicembler generates error-free assemblies. We also show that Slicembler is much more resistant against high sequencing error rate than the base assembler.Availability and implementation: Slicembler can be accessed at http://slicembler.cs.ucr.edu/.Contact: hamid.mirebrahim@email.ucr.edu

show abstract

Section: Introductionmentioning

confidence: 99%

De novo meta-assembly of ultra-deep sequencing data

Mirebrahim

Lonardi

2015

Bioinformatics

View full text Add to dashboard Cite

show abstract

“…By randomly merging counters, the CMS provides an accurate frequency estimate for the most frequent items of a data stream. Diginorm [5], Bignorm [28] and NeatFreq [20] are sketching algorithms for genetic data normalization that use the CMS. These methods reduce the size of genetic datasets by eliminating redundant reads.…”

Section: Composable Coresetsmentioning

confidence: 99%

Diversified RACE Sampling on Data Streams Applied to Metagenomic Sequence Analysis

Coleman

Geordie

Chou

et al. 2019

Preprint

View full text Add to dashboard Cite

The rise of whole-genome shotgun sequencing (WGS) has enabled numerous breakthroughs in large-scale comparative genomics research. However, the size of genomic datasets has grown exponentially over the last few years, leading to new challenges for traditional streaming algorithms. Modern petabyte-sized genomic datasets are difficult to process because they are delivered by high-throughput data streams and are difficult to store. As a result, many traditional streaming problems are becoming increasingly relevant. One such problem is the task of constructing a maximally diverse sample over a data stream. In this regime, complex sampling procedures are not possible due to the overwhelming data generation rate. In theory, the best diversity sampling methods are based on a simple greedy algorithm that compares the current sequence with a large pool of sampled sequences and decides whether to accept or reject the sequence. While these methods are elegant and optimal, they are largely confined to the theoretical realm because the greedy procedure is too slow in practice. While there are many methods to identify common elements in data streams efficiently, fast and memory-efficient diversity sampling remains a challenging and fundamental data streaming problem with few satisfactory solutions. In this work, we bridge the gap with RACE sampling, an online algorithm for diversified sampling. Unlike random sampling, which samples uniformly, RACE selectively accepts samples from streams that lead to higher sequence diversity. At the same time, RACE is as computationally efficient as random sampling and avoids pairwise similarity comparisons between sequences. At the heart of RACE lies an efficient lookup array constructed using locality-sensitive hashing (LSH). Our theory indicates that an accept/reject procedure based on LSH lookups is sufficient to obtain a highly diverse subsample. We provide rigorous theoretical guarantees for well-known biodiversity indices and show that RACE can nearly double the Shannon and Simpson indices of a genetic sample in practice, all while using the same resources as random sampling. We also compare RACE against Diginorm and coreset-based diversity sampling methods and find that RACE is faster and more memory efficient. Our algorithm is straightforward to implement, easy to parallelize, and fast enough to keep pace with the overwhelming data generation rates. We expect that as DNA sequence data streams become more mainstream and faster, RACE will become an essential component for many applications. 1

show abstract

“…NeatFreq [15] clusters and selects short reads based on the median k-mer frequency. However, the main innovation in the work is the inclusion of methods for the use of paired reads alongside with preferential selection of regions with extremely low coverage.…”

Section: Related Workmentioning

confidence: 99%

Improving Metagenomic Assemblies Through Data Partitioning: a GC content approach

Miranda

Batista

Silva

et al. 2018

Preprint

View full text Add to dashboard Cite

Assembling metagenomic data sequenced by NGS platforms poses significant computational challenges, especially due to large volumes of data, sequencing errors, and variations in size, complexity, diversity and abundance of organisms present in a given metagenome. To overcome these problems, this work proposes an open-source, bioinformatic tool called GCSplit, which partitions metagenomic sequences into subsets using a computationally inexpensive metric: the GC content. Experiments performed on real data show that preprocessing short reads with GCSplit prior to assembly reduces memory consumption and generates higher quality results, such as an increase in the N50 metric and the reduction in both the L50 value and the total number of contigs produced in the assembly. GCSplit is available at https://github.com/ mirand863/gcsplit.

show abstract

NeatFreq: reference-free data reduction and coverage normalization for De Novosequence assembly

Cited by 18 publications

References 26 publications

De novo meta-assembly of ultra-deep sequencing data

De novo meta-assembly of ultra-deep sequencing data

Diversified RACE Sampling on Data Streams Applied to Metagenomic Sequence Analysis

Improving Metagenomic Assemblies Through Data Partitioning: a GC content approach

Contact Info

Product

Resources

About