Compression of double array structures for fixed length keywords

Fuketa, Masao; Kitagawa, Hiroya; Ogawa, Takuki; Morita, Ken; Aoe, Jun-ichi

doi:10.1016/j.ipm.2014.04.004

Cited by 16 publications

(16 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The paper by Maeda explained that BASE was deleted by storing BASE values in NEXT. This means BASE is deleted by s=BASE [s] in the same manner of Fuketa"s method [12]. Therefore, (1) …”

Section: A Data Structuresmentioning

confidence: 99%

Comparisons of Efficient Implementations for DAWG

Fuketa¹,

Morita²,

Aoe³

2016

IJCTE

Self Cite

View full text Add to dashboard Cite

Abstract-Key retrieval is very important in various applications. A trie and DAWG are data structures for key retrieval. The double array is one of methods to construct a trie and has both high speed and compactness. In this paper, a data structure of DAWG by the double array using BASE and CHECK is compared with that of DAWG by the double array using CHECK and NEXT, and the retrieval speed and the space usage are theoretically observed. When DAWG and DFA by the double array are constructed, it turns out that it is important to consider indexes for CHECK and NEXT arrays as edge numbers.Index Terms-Automaton, DAWG, double array, triple array. I. INTRODUCTIONKey retrieval is used in various applications [1]. A trie is one of data structures to retrieve keys and merge common prefixes of keys [2]. Moreover, Directed Acyclic Word Gragh (DAWG) is a data structure to reduce the number of trie states [3]. DAWG merges common parts of keys. As the trie and DAWG are kinds of Deterministic Finite Automaton (DFA), they can be traditionally represented by a matrix form (transition table) and a linked list.The triple array is a data structure to construct DFA [4]. This method uses three one-dimensional arrays called BASE, CHECK, and NEXT in order to compress the matrix form. It has high speed because keys of length k can be retrieved by O(k). There is also a method called the double array [5], which compresses the triple array. This method deletes a NEXT array from the triple array and consists of BASE and CHECK. The double array is used in various applications and fields because of its high speed [6], [7]. However, because one state has only one parent state, the original double array can construct a trie but cannot construct DAWG and DFA. Moreover, the compact double array was proposed as a method to compress the double array [8]. This method reduces the space usage by storing traversed characters in CHECK. A method to construct DAWG by the features of the compact double array was proposed [9]. This method has higher speed and less space usage than other methods such as the matrix form, linked list and TST [10]. Furthermore, a method to construct DFA with CHECK and NEXT by deleting a BASE array from the triple array was proposed [11].In this paper, a data structure of DAWG by the double array using BASE and CHECK(BC DAWG) is compared with that of DAWG by the double array using CHECK and NEXT(CN DAWG), and the retrieval speed and the space usage are theoretically observed. A construction algorithm of CN DAWG is proposed. Moreover, features of DAWG by the double array are discussed. II. DOUBLE ARRAY AND DAWGA. Trie A trie is a tree structure used for key retrieval in the field of natural language processing, and is a kind of DFA. Fig. 1 shows examples of the trie in key set K = {"aaa", "aba", "bbc", "cbc", "cc"}. Double circles show terminal states. The trie merges common prefixes of keys. Retrieval always starts from a root state (for example, state number 1 in Fig. 1), and traverses states by one-by-one character in the key....

show abstract

“…The paper by Maeda explained that BASE was deleted by storing BASE values in NEXT. This means BASE is deleted by s=BASE [s] in the same manner of Fuketa"s method [12]. Therefore, (1) …”

Section: A Data Structuresmentioning

confidence: 99%

Comparisons of Efficient Implementations for DAWG

Fuketa¹,

Morita²,

Aoe³

2016

IJCTE

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because next 2-byte of the input key is "b#" that contains the end-mark, the for loop is finished. In (nr-7), t is set to 10 (=BASE [3]+CODE("b#")), and CHECK [10]=3 is satisfied. Because the last traversal is succeeded, this algorithm returns TRUE in (nr-9).…”

Section: B N-gram Double Arraymentioning

confidence: 99%

“…An edge of the trie is represented by a character in previous researches related to the double array. As for the compression of the double array, there are methods dividing the trie [8], [9], a method removing BASE array [10], but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the double array method which represents an edge by n-byte.…”

mentioning

confidence: 99%

A Retrieval Method for Double Array Structures by Using Byte N-Gram

Fuketa¹,

Morita²,

Aoe³

2014

IJTEF

Self Cite

View full text Add to dashboard Cite

Abstract-Retrieving keywords requires speed and compactness. A trie is one of the data structure to retrieve keywords, and the double array is one of the implementation methods for the trie. The retrieval algorithm for the double array is fast, and its data structure has compactness. An edge of the trie is represented by a character in previous researches related to the double array, but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the data structure and the retrieval algorithm for the double array which represents an edge by n-byte. This paper also proposes a method to compress CODE array. From the experimental results comparing with the original double array by using single-byte and multi-byte character sets, the size and the retrieval speed of the proposed method became 62-64% and 1.18-1.3 times, respectively. When the CODE is compressed, the sizes of the proposed method became 41-59%.Index Terms-Compression, double array, n-gram, trie. I. INTRODUCTIONIn ubiquitous environments such as smart phones and PDAs, the storage capacity is often limited. Retrieving keywords used in many applications requires speed and compactness. A trie is one of data structures to retrieve keywords. In the trie, common prefixes of stored keys are merged and each edge is labeled with a character consisting of keys. Because the trie can retrieve common prefix keywords and predictive keywords, it is used in information retrieval systems [1], natural language processing [2], IP address routing tables [3], and packet filtering [4]. Moreover, the trie is often used as an associative array [5] like a map class in C++ in order to improve the array by hash tables.A double array is one of the retrieval methods by using the trie. This method uses two arrays called BASE and CHECK, and it has speed and compactness [6], [7]. An edge of the trie is represented by a character in previous researches related to the double array. As for the compression of the double array, there are methods dividing the trie [8], [9], a method removing BASE array [10], but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the double array method which represents an edge by n-byte. This paper also proposes a method to compress CODE array by using the double array, because CODE array becomes big with n"s increasing.Section II describes the trie and the double array. Section Manuscript received September 23, 2013; revised November 25, 2013. This work was supported by JSPS KAKENHI Grant Number 24500118.The authors are with the Department of Information Science and Intelligent, University of Tokushima, Tokushima, Japan (e-mail: fuketa@is.tokushima-u.ac.jp, kam@is.tokushima-u.ac.jp, aoe@is.tokushima-u.ac.jp).III describes the proposed data structures and retrieval algorithms. Experimental evaluations are given in Section IV. Finally, Section V concludes the proposed algorithm and describes further works. II. DOUBLE ARRAYA trie is a tree structure to store some keys. T...

show abstract

“…Most studies of static DA tries attempted to improve space efficiency and retrieval speed. [8][9][10][11][12] Static DA tries are typically used for dictionaries without frequent key updates (ie, insertions and deletions), such as Darts 13 and Darts-clone. 14 With respect to dynamic DA tries, there are 2 main objectives.…”

Section: Introductionmentioning

confidence: 99%

Practical rearrangement methods for dynamic double‐array dictionaries

et al. 2017

Self Cite

View full text Add to dashboard Cite

SummaryDouble-array structures have been widely used to implement dictionaries with string keys. Although the space efficiency of dynamic double-array dictionaries tends to decrease with key updates, we can still maintain high efficiency using existing methods. However, these methods have practical problems of time and functionality. This paper presents several efficient rearrangement methods to solve these problems.Through experiments using real-world datasets, we demonstrate that the proposed rearrangement methods are much more practical than existing methods.

show abstract

Compression of double array structures for fixed length keywords

Cited by 16 publications

References 11 publications

Comparisons of Efficient Implementations for DAWG

Comparisons of Efficient Implementations for DAWG

A Retrieval Method for Double Array Structures by Using Byte N-Gram

Practical rearrangement methods for dynamic double‐array dictionaries

Contact Info

Product

Resources

About