n-Gram-Based Text Compression

Abstract

We propose an efficient method for compressing Vietnamese text using n-gram dictionaries. It has a significant compression ratio in comparison with those of state-of-the-art methods on the same dataset. Given a text, first, the proposed method splits it into n-grams and then encodes them based on n-gram dictionaries. In the encoding phase, we use a sliding window with a size that ranges from bigram to five grams to obtain the best encoding stream. Each n-gram is encoded by two to four bytes accordingly based on its corresponding n-gram dictionary. We collected 2.5 GB text corpus from some Vietnamese news agencies to build n-gram dictionaries from unigram to five grams and achieve dictionaries with a size of 12 GB in total. In order to evaluate our method, we collected a testing set of 10 different text files with different sizes. The experimental results indicate that our method achieves compression ratio around 90% and outperforms state-of-the-art methods.

Efficiency Analysis of Compression Software (WINRAR and 7-Zip) Across Diverse Data Types on Windows 11 and Ubuntu 23.10

“…Based on the proprietary RAR compression algorithm developed by Eugene Roshal, WinRAR may be considered superior to 7-Zip due to its exceptional compression capabilities. The RAR algorithm employs a combination of Huffman coding, run-length encoding, and adaptive coding to exploit data structures and patterns effectively, resulting in consistently high compression rates [16]. The emphasis on Huffman coding, a key component of the RAR algorithm, signifies a meticulous approach to compression [17].…”

Intelligent RFQ Summarization Using Natural Language Processing, Text Mining, and Machine Learning Techniques

“…In the transformer knowledge domain, many terminologies are composed of two or more words (e.g., oil temperature indicator). N-gram represents a contiguous sequence of n items from a given text or speech (Koehn, 2009;Nguyen, 2016). An n-gram of one item, two items, or three items is referred to as a "unigram," a "bigram," or a "trigram," respectively.…”

Topic Modeling on Online News.Portal Using Latent Dirichlet Allocation (LDA)

“…'N-gram is a contiguous sequence of items from a given line of a text or speech [11]. Ngram is a method that is applied for word or character generation.…”