Development of word-based text compression algorithm for Indonesian language document

Sinaga, Ardiles; Adiwijaya, Adiwijaya; Nugroho, Hary

doi:10.1109/icoict.2015.7231466

Cited by 12 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Stemming dapat diterapkan sebagai text processing, seperti information retrieval, pengecekan plagiarism, peningkatan performa pencarian, dan lainnya [3,4,5,6]. Stemming pada kata tidak baku dapat dimanfaatkan sebagai bagian dari proses sentiment analysis, chat bot, dan pemrosesan teks lainnya [2].…”

Section: Iunclassified

Optimalisasi Stemming Kata Berimbuhan Tidak Baku Pada Bahasa Indonesia Dengan Levenshtein Distance

Putra

Utami

Raharjo

2018

jpit

View full text Add to dashboard Cite

Stemming algorithm Nazief & Andriani has been development in terms of the speed and the accuracy. One of its development is Non-formal Affix Algorithm. Non-formal Affix Algorithm improves the accuracy for non-formal affixed word. In its growth, Indonesian language is used in two ways: formal and non-formal. Non-formal language is commonly used in casual situations such as conversations and social media post (Facebook, Twitter, Instagram, etc.). To get the root of the word of a casual conversation or a social media post, stemming algorithm which can process the non-formal words with affixes already proposed. But, the previous algorithm unable to stem a non-formal word that slightly change the root word. Therefore, this study modifies Non-formal Affix Algorithm to increase stemming accuracy on non-formal word. Modifications are made by adding Levenshtein Distance. The result of the research shows that the algorithm made in this research has 96.6% accuracy while the Non-formal Affix algorithm has 73.3% accuracy in processing 60 non-formal affixed words. Based on the result, Levenshtein Distance approach can increase the accuracy on stemming non-formal affixed word.

show abstract

Section: Iunclassified

Optimalisasi Stemming Kata Berimbuhan Tidak Baku Pada Bahasa Indonesia Dengan Levenshtein Distance

Putra

Utami

Raharjo

2018

jpit

View full text Add to dashboard Cite

show abstract

“…Penelitian ini menjadi dasar algoritma stemming pada Bahasa Indonesia yang cukup popular, yaitu Confix Stripping (ECS Stemmer) [8]. Penelitian lain berupaya menemukan aturan morfologi secara otomatis melalui proses learning [9], memodifikasi stemming untuk proses lematisasi [10] atau menemukan manfaat lain dari proses stemming, misalnya untuk kompresi teks [11] dan untuk mendeteksi kesalahan pengetikan (typo) [12].…”

Section: Pendahuluanunclassified

Untitled

Hidayat

2017

INFOTEL

View full text Add to dashboard Cite

Abstrak -Ekstraksi kata dasar atau stemming pada Bahasa Indonesia adalah proses yang kompleks di mana beberapa partikel awal dan beberapa partikel akhiran dari 13 awalan, 3 sisipan, dan 19 akhiran yang dikenal dapat digunakan secara sekaligus pada sebuah kata. Selain itu, proses stemming tidak terlalu menghasilkan 1 kata dasar (non-deterministik) karena terdapat beberapa kata dalam Bahasa Indonesia yang memiliki 2 kemungkinan, yaitu sebagai kata dasar maupun kata berimbuhan, misalnya pada kata "beruang". Penelitian yang telah ada sebelumnya menggunakan kombinasi awalan dan akhiran yang tidak mungkin dan menerapkan heuristik untuk memilih kata dasar. Dalam penelitian ini diusulkan sebuah metode stemming secara berjenjang di mana berdasarkan urutan tertentu, secara bergantian partikel akhiran dan awalan dilepaskan dari sebuah kata sehingga dihasilkan sebuah kata dasar. Jika ditemukan beberapa kandidat kata dasar maka salah satu kata dasar akan dipilih. Metode ini diuji pada 6464 dokumen Al-Quran Terjemahan Indonesia dengan menggunakan kamus berukuran 5000 kata yang dipilih secara acak dari Kamus Besar Bahasa Indonesia. Dari 3432 kata unik yang diproses, diperoleh 94,7% kata dasar yang dapat diekstrak secara langsung dan hanya 5,3% yang perlu diproses lebih lanjut karena kandidat kata dasar yang ditemukan lebih dari satu. Dibandingkan dengan melakukan pemilihan kata dasar secara manual, metode ini dapat memilih kata dasar yang tepat hingga 79,12%.Kata kunci -ekstraksi kata dasar, morfologi, non-deterministik, akurasi Abstract-Stemming in Indonesian Language is a complex process where few particles among 13 known prefixes, 3 known infixes, and 19 known suffixes can be used simultaneously in a single word. Moreover, stemming process does not guarantee a single word result (hence, non-deterministic), because there are few words in Indonesian Language that has 2 possibilities, either it is a morpheme itself or it is an affixed word, e.g. the word "beruang". Previous researches have tried to deal with this issue by listing impossible prefix and suffix combination while at the same time implementing heuristic to identify final stem. In this paper, we present an incremental stemming method, where based on particular order, suffixes and prefixes are alternatingly stripped until a single morpheme (stem) is found. In case there are several stem candidates were found, then one stem will be chosen. This method is tested against 6464 text files from Holy Quran Indonesian Translation using 5000-word dictionary randomly chosen from Kamus Besar Bahasa Indonesia. From 3432 unique words, 94.7% single stem can be identified immediately and only 5.3% that need to be processed further since there are more than one candidate stem were found. Compared to manually stem selection, this method is able to accurately choose 79.12% correct stem.

show abstract

“…Some word-based compression exploits each word considering a basic unit [4]. Using Huffman coding, wordbased text compression has already raised a standard [11]. Many natural language text compressors used data compression techniques for general purposes by performing the character-level compression [12].…”

Section: Introductionmentioning

confidence: 99%

Method of Adjacent Distance Array Outperforms Conventional Huffman Codes to Decode Bengali Transliterated Text Swiftly

Sarker¹,

Rahman²

2022

IJCDS

View full text Add to dashboard Cite

This research works on high symbolic Bengali text and transforms it into corresponding less symbolic English complying with the transliteration method. The Huffman-based approaches serve to compress retaining the original quality of the data. On the other hand, faster encoding and decoding is the most sophisticated sphere in data compression. We propose an adjacent distance array, a novel data structure based on the Huffman principle for encoding and decoding the character of transliterated text. The encoding and decoding algorithms have been explained for the introduced modus operandi and juxtaposed with conventional Huffman-based algorithms. Our research is outdoing than any regular Huffman-based algorithms, concentrating on the speed of the encoding and decoding manner discovered after estimating all decisions.

show abstract

Development of word-based text compression algorithm for Indonesian language document

Cited by 12 publications

References 6 publications

Optimalisasi Stemming Kata Berimbuhan Tidak Baku Pada Bahasa Indonesia Dengan Levenshtein Distance

Optimalisasi Stemming Kata Berimbuhan Tidak Baku Pada Bahasa Indonesia Dengan Levenshtein Distance

Untitled

Method of Adjacent Distance Array Outperforms Conventional Huffman Codes to Decode Bengali Transliterated Text Swiftly

Contact Info

Product

Resources

About