Efficient Construction of a Complete Index for Pan-Genomics Read Alignment

Abstract

Short-read aligners predominantly use the FM-index, which is easily able to index one or a few human genomes. However, it does not scale well to indexing collections of thousands of genomes. Driving this issue are the two chief components of the index: (1) a rank data structure over the Burrows-Wheeler Transform (BWT) of the string that will allow us to find the interval in the string's suffix array (SA), and (2) a sample of the SA that-when used with the rank data structure-allows us to access the SA. The rank data structure can be kept small even for large genomic databases, by run-length compressing the BWT, but until recently there was no means known to keep the SA sample small without greatly slowing down access to the SA. Now that (SODA 2018) has defined an SA sample that takes about the same space as the run-length compressed BWT, we have the design for efficient FM-indexes of genomic databases but are faced with the problem of building them. In 2018, we showed how to build the BWT of large genomic databases efficiently (WABI 2018), but the problem of building the sample efficiently was left open. We compare our approach to state-of-the-art methods for constructing the SA sample, and demonstrate that it is the fastest and most space-efficient method on highly repetitive genomic databases. Lastly, we apply our method for indexing partial and whole human genomes and show that it improves over the FMindex-based Bowtie method with respect to both memory and time and over the hybrid index-based CHIC method with respect to query time and memory required for indexing.

Reference flow: reducing reference bias using multiple population genomes

“…This might be accomplished using unsupervised, sequence-driven clustering methods [ 34 , 35 ], using the “founder sequence” framework [ 36 , 37 ], or using some form of submodular optimization [ 38 ]. A more radical idea is to simply index all available individuals, forgoing the need to choose representatives; this is becoming more practical with the advent of new approaches for haplotype-aware path indexing [ 31 ] and efficient indexing for repetitive texts [ 39 ].…”

Computational graph pangenomics: a tutorial on data structures and their applications

“…As previously mentioned, Gagie et al (2020) did not describe how to build the r-index -this was shown in a series of papers (Kuhnle et al 2020;Mun et al 2020;Boucher et al 2019). In particular, Boucher et al (2019) introduced Prefix Free Parsing (PFP), which takes as input a string S, window size w, and a prime p and produces a dictionary of substrings of S and a parse of S, that is a sequence of substrings in the alphabet (Kreft and Navarro 2013) -and showed how to build RLBWT from the dictionary and parse.…”

Next Generation Sequencing Technology in the Clinic and Its Challenges

“…Briefly explained, FM-index alignment tools are derived from the Burrows-Wheeler Transform [ 68 ]—a method to sufficiently compress large amount of data and finding approximate matches of sequences in the reference genome [ 69 ]. Hash table-based aligners uses the seed-and-extend method in combination with additional alignment algorithms [ 68 , 70 , 71 ].…”