A New Feature Selection Method for Sentiment Analysis in Short Text

Abstract

<jats:title>Abstract</jats:title> <jats:p>In recent internet era, micro-blogging sites produce enormous amount of short textual information, which appears in the form of opinions or sentiments of users. Sentiment analysis is a challenging task in short text, due to use of formal language, misspellings, and shortened forms of words, which leads to high dimensionality and sparsity. In order to deal with these challenges, this paper proposes a novel, simple, and yet effective feature selection method, to select frequently distributed features related to each class. In this paper, the feature selection method is based on class-wise information, to identify the relevant feature related to each class. We evaluate the proposed feature selection method by comparing with existing feature selection methods like chi-square ( <jats:italic>χ</jats:italic><jats:sup>2</jats:sup>), entropy, information gain, and mutual information. The performances are evaluated using classification accuracy obtained from support vector machine, K nearest neighbors, and random forest classifiers on two publically available datasets viz., Stanford Twitter dataset and Ravikiran Janardhana dataset. In order to demonstrate the effectiveness of the proposed feature selection method, we conducted extensive experimentation by selecting different feature sets. The proposed feature selection method outperforms the existing feature selection methods in terms of classification accuracy on the Stanford Twitter dataset. Similarly, the proposed method performs competently equally in terms of classification accuracy compared to other feature selection methods in most of the feature subsets on Ravikiran Janardhana dataset.</jats:p>

Improving Arabic Cognitive Distortion Classification in Twitter using BERTopic

“…The limited number of words in tweets leads to sparse cooccurrence patterns, making the classification task more challenging. To tackle the feature sparseness, works in the literature choose between two main approaches; either represent texts in a lower-dimensional space [9] or add external, implicit, and valuable information to enhance the data representation on the feature space [10], [11]. The second method works on enriching the representation of a short text using additional knowledge or semantics [15].…”

“…This work aims to improve the classification task by dealing with the text shortness problem. In general, many classification tasks working with short text fails to achieve high performance according to the sparse representation of the textual data [9], [10], [11]. Extra contextual information can be deployed to overcome the sparse representation and make the data more related, comprehensively expressed, and expand the efficiency of classifiers to handle unseen data.…”

A Comparison of Sentimental Analysis Algorithms on Twitter Data Using Machine Learning

“…Geetika Gautam and Divakar Yadav [5], on the basis of Twitter data, for detecting sentences and product reviews, a set of machine learning approaches with semantic analysis was presented. The main purpose is to use a pre-labeled Twitter dataset to analyse a large number of reviews.…”

A deep learning approach for the depression detection of social media data with hybrid feature selection and attention mechanism

“…However, the semantic‐based models perform poorly due to the nature of an opinionated text requiring higher understanding for text processing. Several studies have found that ML‐based approaches outperform lexicon‐based methods in DD prediction (Keerthi Kumar & Harish, 2020). ML techniques aid in the faster detection of disorders with high AC and a low misclassification rate.…”