Categorization of Malay Social Media Text and Normalization of Spelling Variations and Vowel-less Words

Abstract

As more data are being introduced, it brings along with it missing values, inconsistencies, and heterogeneities, or so-called unclean aspects. Text analytics relies on clean data to produce reliable results. Pre-processing is an essential phase in text analytics, specifically language detection and normalization. The problem with conducting text analytics on Malay social media text is how substantially it has transformed from formal Malay in terms of spelling and construction, making it difficult to process them. Recent advances have shown works to normalize yet cherry-picked specific types of Malay social media text where their descriptions were listed in simple and narrow categorizations. A formal categorization is necessary to provide significant description of the different patterns of Malay social media text, allowing the selection of suitable methods in handling them. In this paper, we propose an inexhaustive formal categorization for Malay social media text based on inherent nature. We refer to them as Social Media Malay Language (SMML) to differentiate them from the standard Malay language. They are spelling variations, Malay-English mix sentences, loan words/phrases, slang-based words, and vowel-less words. Also, in this work, we conducted a normalization on two of the SMML categories, spelling variations, and vowel-less words, using two similarity matching techniques (i.e., nGram Tversky Index and Levenshtein). Our result shows that similarity-matching techniques can detect both categories, but a more sophisticated technique is necessary to improve the precision score. The normalization of the rest of the categories is extensive research works.

Non-Conventional Spelling in Informal, Colloquial Arabic Writing on Facebook

“…In Nigeria, use of social media had a negative effect on college students' spelling ability and the conventional way of writing in examinations and letters (Wilson, 2018). In Malaysia, some of the weaknesses found in Malay social media texts were spelling variations, and vowel-less words (Maskat & Rahman, 2020). Use of pairs of word forms covering seven types of spelling variation in English was found on Twitter and Reddit (Nguyen, Grieve, Scott, Bel & Zong, 2020).…”

“…The researcher found that use of social media had a negative effect on college students' spelling ability and the conventional way of writing especially in examinations and letters. Some of the weaknesses found in Malay social media texts were spelling variations, and vowel-less words, Malay-English mix in sentences, loan words/phrases, and slang-based words (Maskat & Rahman, 2020).…”

A Systematic Literature Review on Social Media Slang Analytics in Contemporary Discourse

“…Grasping user perspectives assumes a pivotal role in fostering precise sentiment analysis within the context of social media slang analytics, as it facilitates the interpretation of subtle slang expressions and their associated connotations in online interactions. Additionally, numerous studies have predominantly focused on widely recognized social media platforms such as Twitter [7] [12] [13]. However, an extensive assessment of the diversity encompassing platforms and data sources employed in slang analytics is conspicuously absent.…”