Slang Detection and Identification

Abstract

The prevalence of informal language such as slang presents challenges for natural language systems, particularly in the automatic discovery of flexible word usages. Previous work has explored slang in terms of dictionary construction, sentiment analysis, word formation, and interpretation, but scarce research has attempted the basic problem of slang detection and identification. We examine the extent to which deep learning methods support automatic detection and identification of slang from natural sentences using a combination of bidirectional recurrent neural networks, conditional random field, and multilayer perceptron. We test these models based on a comprehensive set of linguistic features in sentencelevel detection and token-level identification of slang. We found that a prominent feature of slang is the surprising use of words across syntactic categories or syntactic shift (e.g., verb→noun). Our best models detect the presence of slang at the sentence level with an F1-score of 0.80 and identify its exact position at the token level with an F1-Score of 0.50.

A Computational Framework for Slang Generation

“…In their study, both the spelling of a word and its context are provided as input to a translation model to decode a definition sentence. Pei et al (2019) proposed end-to-end neural models to detect and identify slang automatically in natural sentences. Kulkarni and Wang (2018) have proposed computational models that derive novel word forms of slang from spellings of existing words.…”

“…Here, we focus on syntax and linguistic context, although our framework should allow for the incorporation of social or extra-linguistic features as well. Recent work has found that the flexibility of slang is reflected prominently in syntactic shift (Pei et al, 2019). For example, ice-most commonly used as a noun-is used as a verb to express ''to kill'' (in Figure 1).…”

IndoCollex: A Testbed for Morphological Transformation of Indonesian Word Colloquialism

“…Wikipedia) to transfer to another (e.g. social media) (Eisenstein, 2013b;Baldwin et al, 2013b;Belinkov and Bisk, 2018;Pei et al, 2019). Worse yet, for an overwhelming majority of lower resource languages, unstructured and unlabeled text on the Internet is often the sole source of data to train NLP systems (Joshi et al, 2020).…”

“…Most of the dataset: formal-informal word pairs labeled with their word formation used to train these models are also in English. Other dictionaries of informal English words include SlangNet (Dhuliawala et al, 2016), SlangSD (Wu et al, 2018), and SLANGZY (Pei et al, 2019). There is also a dataset that contains pairs of formal-informal Indonesian words (Salsabila et al, 2018), but they are not annotated with word formation mechanisms.…”

“…In Korean, some compounded or shortened version of Konglish is also widely used (Khan and Choi, 2016), e.g., chimaek from chicken and maek ('beer'). Any insight we obtain through evaluating models on our dataset may therefore be of interest to other languages that share similar colloquial transformations; insights that may be increasingly paramount due to the rising prevalance of non-standard text in many languages on the web (Kulkarni and Wang, 2018;Joshi et al, 2020) and the challenges they pose to NLP systems (Belinkov and Bisk, 2018;Pei et al, 2019).…”