Normalization of Unstructured and Informal Text in Sentiment Analysis

Abstract

Sentiment Analysis is problem of natural language processing which deals with the extraction and analysis of public sentiments shared about target entities over microbloging websites. This field has gained great attention due to the huge availability of decision making textual contents. Sentiment Analysis has enormous application areas such as; Market Analysis, Service Analysis, Showbiz analysis, Movies, sports and even the popularity and acceptance rate of political policies can also be predicted via sentiment analysis systems. Although tremendous volume of opinionative text is available but it is unstructured and noisy due to which sentiment classifiers can't achieve good outcomes. Normalization is the process used to clean noise from unstructured text for sentiment analysis. In this study we have proposed a mechanism for the normalization of informal and unstructured text. Proposed mechanism is comprised of four essential phases; Noise Reduction, Part of Speech Tagging, Stop Word Removal stemming and Lemmatization. Numerous experiments are performed on twitter data set with unsupervised lexicons and dictionaries. Python and Natural language toolkit is used for performing all four essential steps. This study demonstrates that utilization and normalization of informal tokens in tweets improved the overall classification accuracy from 75.42 to 82.357.

Indonesian Sentiment Analysis towards MyPertamina Application Reviews by Utilizing Machine Learning Algorithms

“…After obtaining data, the next step is data normalization. The purpose of this phase is to normalize the review since some reviews contains slang words or clean the data from the noise [8]. This work has been done using Indonesian colloquial words collection [9].…”

Sentimen Analisis Terhadap Aplikasi pada Google Playstore Menggunakan Algoritma Naïve Bayes dan Algoritma Genetika

“…Translate berguna untuk mengganti kata yang berbahasa asing kedalam bahasa indonesia [4]. Dan terakhir adalah normalisasi yang berfungsi untuk menghilangkan noise seperti merubah kata tidak baku menjadi baku [11]. Kemudian masuk ketahap klasifikasi menggunakan algoritma naïve bayes, hal pertama yang dilakukan adalah mencari nilai probability masing-masing kata dataset menggunakan persamaan matematik tertentu seperti pada persamaan 2.…”

Support Vector Machine Untuk Identifikasi Berita Hoax Terkait Virus Corona (Covid-19)

“…Penghapusan stopwords bertujuan untuk mengurangi skala indeks data yang nantinya berpengaruh pada kecepatan dan peforma dari sistem yang dibuat. Setelah penghapusan "Stopwords", Langkah selanjutnya adalah proses Stemming, pada proses stemming ini suatu kata akan diproses kembali untuk menghilangkan prefiks dan sufiksnya sehingga kata tersebut menjadi kata dasar sehingga nantinya dapat meningkatkan kinerja pencarian kata [24].…”