Suffix Array Construction in External Memory Using D-Critical Substrings

Abstract

We present a new suffix array construction algorithm that aims to build, in external memory, the suffix array for an input string of length n measured in the magnitude of tens of Giga characters over a constant or integer alphabet. The core of this algorithm is adapted from the framework of the original internal memory SA-DS algorithm that samples fixed-size d-critical substrings. This new external-memory algorithm, called EM-SA-DS, uses novel cache data structures to construct a suffix array in a sequential scanning manner with good data spatial locality: data is read from or written to disk sequentially. On the assumed external-memory model with RAM capacity Ω (( nB ) 0.5 ), disk capacity O ( n ), and size of each I/O block B , all measured in log n -bit words, the I/O complexity of EM-SA-DS is O ( n / B ). This work provides a general cache-based solution that could be further exploited to develop external-memory solutions for other suffix-array-related problems, for example, computing the longest-common-prefix array, using a modern personal computer with a typical memory configuration of 4GB RAM and a single disk.

Induced Sorting Suffixes in External Memory

“…dsa[ j] may be currently on disk and will be accessed slowly. To overcome this difficulty, the blockwise-induced sorting method reported in Beller et al [2013] and Nong et al [2014] can be applied to avoid random access of dsa [ j]. Blockwise-induced sorting evenly splits dsa into a number of blocks (the last block may be smaller).…”

“…Recently, work has been done on external memory SACAs. Recent experimental results have shown that algorithms based on the induced sorting principle, that is, eSAIS [Bingmann et al 2013] and EM-SA-DS [Nong et al 2014], are competitive against the previously best techniques such as DC3 [Dementiev et al 2008] and bwtdisk [Ferragina et al 2012]. 1 Let SA(x) denote the suffix array of x.…”

“…Retrieving x[ j] without random accesses of x is still a challenge in the existing external memory designs for induced sorting suffixes. The attempts to solve this problem in Bingmann et al [2013] and Nong et al [2014] resulted in much more complicated designs than their counterparts in the RAM.…”

“…Following our recent work [Nong et al 2014] developing an external memory design for the algorithm SA-DS [Nong et al 2011], we propose a new solution called disk SA-IS (DSA-IS) in this article. Pleasingly, unlike the solutions in Bingmann et al [2013] and Nong et al [2014], DSA-IS emulates SA-IS merely by replacing random accesses of the RAM with sequential accesses of the external memory.…”

“…Pleasingly, unlike the solutions in Bingmann et al [2013] and Nong et al [2014], DSA-IS emulates SA-IS merely by replacing random accesses of the RAM with sequential accesses of the external memory. We develop DSA-IS on the same settings as the external memory model in Nong et al [2014] with the following memory parameters (in log n-bit words): RAM capacity M = ((nW ) 0.5 ), disk capacity E = O(n), and the size of each I/O buffer W. W is large enough to amortize the overhead of each disk access; given W, the I/O complexity in terms of the number of I/Os is defined as the total I/O volume over W.…”