Enhancing recurrent neural network-based language models by word tokenization

Abstract

Statistical language models estimate the probability for a given sequence of words. Given a sentence s with n words such as s = (w 1 , w 2. .. w n), the language model assigns P(s). Statistical language models assess good word sequence estimations based on the sequence probability estimation. Building robust, fast and accurate language models is one of the main factors in the success of building systems such as machine translation systems and automatic speech recognition systems. Statistical language models can be estimated based on various approaches. Classical language models are estimated based on n-gram word sequences such as P(s) = P(w 1 , w 2. .. w n) = n i=1 P(w i |w i−1) and can be approximated based on the Markov concept for shorter contexts (for example, bigram if n = 2 or trigram if n = 3 and so on). Recent researchers have applied neural networks different architectures to build and estimate language models. The classical feed-forward neural network-based language models have been continuously reporting good results among the traditional n-gram language modeling techniques [1]. It

Aplikasi Pendeteksi Kesamaan Dokumen Dengan Menggunakan Algoritma Jarak Jaro Winkler Dan Levenshtein

“…Tokenisasi kata-kata tersebut menjadi tiga bagian: bagian awalan, bagian batang, dan bagian akhiran. Model pada penelitian ini akan diuji dengan menggunakan dataset pengenal ucapan AMI bahasa Inggris serta melebihi model n-gram awal (Noaman, 2018) 2. Case Folding Pada tahap ini hasil dari tokenisasi diubah menjadi huruf kecil atau tidak huruf besar semua.…”

Lexicon - pointed hybrid N-gram Features Extraction Model (LeNFEM) for Sentence Level Sentiment Analysis.

“…Social media texts are normally unstructured, noisy and inconsistent. 29 The tweets are cleaned of non-English words or sentences, abbreviations and emoticons and stop words after input. Tokenization and transformation of the texts into lower case is also done to split the sentences into separable words.…”

Algoritma Jaro-Winkler Distance: Fitur Autocorrect dan Spelling Suggestion pada Penulisan Naskah Bahasa Indonesia di BMS TV

“…Sebelum proses pengukuran jarak, kata-kata atau kalimat pengguna diolah pada tahap preprocessing terlebih dahulu Mendeteksi token dan batas kalimat adalah langkah preprocessing penting dalam aplikasi pemrosesan bahasa alami karena sebagian besar beroperasi baik pada tingkat kata (misalnya, silabus, analisis morfologi) atau kalimat (misalnya pemberian tag bagian-pidato, parsing, terjemahan mesin). Beberapa penelitian tentang pemrosesan bahasa menyertakan proses tokenizing pada tahap preprocessing, (Omar, 2018) melakukan tonizing pada tahap preprocessing untuk memperoleh token bahasa arab, (Noaman, Sarhan dan Rashwan, 2018) menyajikan model bahasa jaringan saraf berdasarkan tokenisasi kata-kata menjadi tiga bagian: awalan, batang, dan akhiran. Model yang diusulkan diuji dengan dataset pengenal ucapan AMI bahasa Inggris dan melebihi model n-gram awal.…”