An Online Algorithm for Lightweight Grammar-Based Compression

Maruyama, Shirou; Takeda, Masayuki; Nakahara, Masaya; Sakamoto, Hiroshi

doi:10.1109/ccp.2011.40

Cited by 13 publications

(14 citation statements)

References 33 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, with more work, our method will make it possible to detect plagiarism from large document collections and interesting patterns from biological sequences using a novel technique in [17]. An important future work is to improve our method to handle the stream data, which is partially developed by an online compression algorithm for grammar-based compression in [18] we recently proposed.…”

Section: Discussionmentioning

confidence: 99%

Scalable Detection of Frequent Substrings by Grammar-Based Compression

Nakahara

Maruyama

Kuboyama

et al. 2013

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

SUMMARYA scalable pattern discovery by compression is proposed. A string is representable by a context-free grammar deriving the string deterministically. In this framework of grammar-based compression, the aim of the algorithm is to output as small a grammar as possible. Beyond that, the optimization problem is approximately solvable. In such approximation algorithms, the compressor based on edit-sensitive parsing (ESP) is especially suitable for detecting maximal common substrings as well as long frequent substrings. Based on ESP, we design a linear time algorithm to find all frequent patterns in a string approximately and prove several lower bounds to guarantee the length of extracted patterns. We also examine the performance of our algorithm by experiments in biological sequences and other compressible real world texts. Compared to other practical algorithms, our algorithm is faster and more scalable with large and repetitive strings.

show abstract

Section: Discussionmentioning

confidence: 99%

Scalable Detection of Frequent Substrings by Grammar-Based Compression

Nakahara

Maruyama

Kuboyama

et al. 2013

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…These methods exploit repetitions in the text to derive good grammar rules, so they are particularly suitable for texts containing many identical substrings. Finding the smallest grammar for a given text is NP-hard [21], but there exist several grammar-based compressors that achieve O(log N ) approximation factors or less [71,73,59,47], where N is the text length. We use Re-Pair [52] as our grammar compressor.…”

Section: Data Compression and Codingmentioning

confidence: 99%

Practical compressed string dictionaries

Martínez‐Prieto

Brisaboa

Cánovas

et al. 2016

Information Systems

View full text Add to dashboard Cite

The need to store and query a set of strings -a string dictionary -arises in many kinds of applications.While classically these string dictionaries have accounted for a small share of the total space budget (e.g., in Natural Language Processing or when indexing text collections), recent applications in Web engines, Semantic Web (RDF) graphs, Bioinformatics, and many others, handle very large string dictionaries, whose size is a significant fraction of the whole data. In these cases, string dictionary management is a scalability issue by itself. This paper focuses on the problem of managing large static string dictionaries in compressed main memory space. We revisit classical solutions for string dictionaries like hashing, tries, and front-coding, and improve them by using compression techniques. We also introduce some novel string dictionary representations built on top of recent advances in succinct data structures and full-text indexes. All these structures are empirically compared on a heterogeneous testbed formed by real-world string dictionaries. We show that the compressed representations may use as little as 5% of the original dictionary size, while supporting lookup operations within a few microseconds. These numbers outperform the state-of-the-art space/time tradeoffs in many cases. Furthermore, we enhance some representations to provide prefix-and substringbased searches, which also perform competitively. The results show that compressed string dictionaries are a useful building block for various data-intensive applications in different domains.

show abstract

“…An important line of research in this regard are the online grammar construction algorithms [Maruyama et al 2012;Takabatake et al 2017].…”

Section: Construction and Dynamismmentioning

confidence: 99%

Indexing Highly Repetitive Collections

Navarro

2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The need to index and search huge highly repetitive sequence collections is rapidly arising in various fields, including computational biology, software repositories, versioned collections, and others. In this short survey we briefly describe the progress made along three research lines to address the problem: compressed suffix arrays, grammar compressed indexes, and Lempel-Ziv compressed indexes.

show abstract

An Online Algorithm for Lightweight Grammar-Based Compression

Cited by 13 publications

References 33 publications

Scalable Detection of Frequent Substrings by Grammar-Based Compression

Scalable Detection of Frequent Substrings by Grammar-Based Compression

Practical compressed string dictionaries

Indexing Highly Repetitive Collections

Contact Info

Product

Resources

About