Identifying domain relevant user generated content through noise reduction: a test in a Chinese stock discussion forum

Abstract

Purpose Getting high-quality data by removing the noisy data from the user-generated content (UGC) is the first step toward data mining and effective decision-making based on ubiquitous and unstructured social media data. This paper aims to design a framework for revoking noisy data from UGC. Design/methodology/approach In this paper, the authors consider a classification-based framework to remove the noise from the unstructured UGC in social media community. They treat the noise as the concerned topic non-relevant messages and apply a text classification-based approach to remove the noise. They introduce a domain lexicon to help identify the concerned topic from noise and compare the performance of several classification algorithms combined with different feature selection methods. Findings Experimental results based on a Chinese stock forum show that 84.9 per cent of all the noise data from the UGC could be removed with little valuable information loss. The support vector machines classifier combined with information gain feature extraction model is the best choice for this system. With longer messages getting better classification performance, it has been found that the length of messages affects the system performance. Originality/value The proposed method could be used for preprocessing in text mining and new knowledge discovery from the big data.

Inducing stock market lexicons from disparate Chinese texts

“…In particular, there has been evidence that the interaction between sentiment expressed by social and conventional media has strong effect on market variables prediction (Yu et al, 2013;Li et al, 2016;Agarwal et al, 2019). Therefore, it is crucial to identify high-quality relevant content and conduct SA on integrated data, extract collective market sentiment and understand their joint influences (Kearney and Liu, 2014;Yan et al, 2017;Li et al, 2018).…”

Unsupervised Creation of Normalization Dictionaries for Micro-Blogs in Arabic, French and English

“…[1] were first to propose that method with a training dataset of 27356 English SMS phrases. His research was the base of several similar work in Portuguese [8], Turkish [9] and Chinese [10], but never in Arabic nor French. In addition, none of these work is open source, and they didn't share the word embedding models, nor the lexicons or dictionaries.…”

“…-For the English language, the 3429 English words from Oxford dictionary 9 which English stop-words are included, the 500 most frequently used words on twitter 10 and a list of 90 frequent sentiment words 11 in tweets [13] were combined. A list of 3501 English words is the result of the previous lists combination, after removing duplications.…”