Text Normalization in Social Media: Progress, Problems and Applications for a Pre-Processing System of Casual English

Clark, Eleanor; Araki, Kenji

doi:10.1016/j.sbspro.2011.10.577

Cited by 114 publications

(73 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These challenges are largely due to the inherent main social data characteristics (summarized in • Vast social data sizes: Demand scalable solutions, dealing with computational time complexities required by conventional text mining algorithms. Emerging clustering approaches should be considered to result in efficient social text data analysis, since data need to be processed at a limited amount of time [12,13] . Current parallel and distributed infrastructures are proposed to meet the scaling demands of the clustering algorithms, and already several parallel or distributed clustering approaches have been proposed reducing both the computational cost and the execution time [12,14,15] .…”

Section: Challenges In Crowdsourced Trend Analysismentioning

confidence: 99%

Cloud-based architectures for geo-located blogosphere dynamics detection

2016

View full text Add to dashboard Cite

Social networking data threads emerge rapidly and such crowd-driven big data streams are valuable for detecting trends and opinions. For such analytics, conventional data mining approaches are challenged by both high-dimensionality and scalability concerns. Here, we leverage on the Cloud4Trends framework for collecting and analyzing geo-located microblogging content, partitioned into clusters under cloud-based infrastructures. Different cloud architectures are proposed to offer flexible solutions for geo-located data analytics with emphasis on incremental trend analysis. The proposed architectures are largely based on a set of service modules which facilitate the deployment of the experimentation on cloud infrastructures. Several experimentation remarks are highlighted to showcase the requirements and testing capabilities of different cloud computing settings. Keywords: social networks and wisdom of the crowd, geo-located blogosphere dynamics, social geo-located data clustering, cloud service deployment.

show abstract

Section: Challenges In Crowdsourced Trend Analysismentioning

confidence: 99%

Cloud-based architectures for geo-located blogosphere dynamics detection

2016

View full text Add to dashboard Cite

show abstract

“…Processing Social Media Text Finally, while English NLP for social media has attracted considerable attention recently (Clark and Araki, 2011;Gimpel et al, 2011;Gouws et al, 2011;Ritter et al, 2011;Derczynski et al, 2013), there has not been much work on Arabic yet. Darwish et al (2012) discuss NLP problems in retrieving Arabic microblogs (tweets).…”

Section: Related Workmentioning

confidence: 99%

Transliteration of Arabizi into Arabic Orthography: Developing a Parallel Annotated Arabizi-Arabic Script SMS/Chat Corpus

Bies¹,

Song²,

Maamouri³

et al. 2014

Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)

View full text Add to dashboard Cite

This paper describes the process of creating a novel resource, a parallel Arabizi-Arabic script corpus of SMS/Chat data. The language used in social media expresses many differences from other written genres: its vocabulary is informal with intentional deviations from standard orthography such as repeated letters for emphasis; typos and nonstandard abbreviations are common; and nonlinguistic content is written out, such as laughter, sound representations, and emoticons. This situation is exacerbated in the case of Arabic social media for two reasons. First, Arabic dialects, commonly used in social media, are quite different from Modern Standard Arabic phonologically, morphologically and lexically, and most importantly, they lack standard orthographies. Second, Arabic speakers in social media as well as discussion forums, SMS messaging and online chat often use a non-standard romanization called Arabizi. In the context of natural language processing of social media Arabic, transliterating from Arabizi of various dialects to Arabic script is a necessary step, since many of the existing state-of-the-art resources for Arabic dialect processing expect Arabic script input. The corpus described in this paper is expected to support Arabic NLP by providing this resource.

show abstract

“…While Sproat et al (2001) work with four different types of text (newspaper text, real estate ads, and servlist texts on the topics of palmtop computers and cooking recipes), most later papers deal either with SMS texts -see for instance Choudhury et al (2007), Kobus et al (2008), or Cook and Stevenson (2009) -or Twitter text -see for instance Clark and Araki (2011), Brody and Diakopoulos (2011), Foster et al (2011), Han and Baldwin (2011), Hassan and Menezes (2013 or Eisenstein (2013). Also, Liu et al (2012) have worked on both SMS and Twitter datasets.…”

Section: Text Normalization As a Taskmentioning

confidence: 99%

“…Zhu et al (2007)'s approach tackles the normalization task as a tagging problem, where the different normalization transformations required are performed on each token depending on its type (line break, space, punctuation, word and special). Clark (2003) and Clark and Araki (2011) use rule-based approaches while Henríquez Q. and Hernández (2009) approach the task using a statistical machine translation system trained on original texts and their semi-automatically corrected version. More recently, Hassan and Menezes (2013) have shown how a version of Markov Random Walks can be used to train a normalizer on a small manually corrected corpus -without any further labeling.…”

Section: Approaches To Text Normalizationmentioning

confidence: 99%

Selection of correction candidates for the normalization of Spanish user-generated content

et al. 2014

View full text Add to dashboard Cite

We present research aiming to build tools for the normalization of User-Generated Content (UGC). We argue that processing this type of text requires the revisiting of the initial steps of Natural Language Processing (NLP), since UGC (micro-blog, blog, and, generally, Web 2.0 user generated texts) presents a number of non-standard communicative and linguistic characteristics -often closer to oral and colloquial language than to edited text. We present a corpus of UGC text in Spanish from three different sources: Twitter, consumer reviews and blogs, and describe its main characteristics. We motivate the need for UGC text normalization by analyzing the problems found when processing this type of text through a conventional language processing pipeline, particularly in the tasks of lemmatization and morphosyntactic tagging.

show abstract

Text Normalization in Social Media: Progress, Problems and Applications for a Pre-Processing System of Casual English

Cited by 114 publications

References 2 publications

Cloud-based architectures for geo-located blogosphere dynamics detection

Cloud-based architectures for geo-located blogosphere dynamics detection

Transliteration of Arabizi into Arabic Orthography: Developing a Parallel Annotated Arabizi-Arabic Script SMS/Chat Corpus

Selection of correction candidates for the normalization of Spanish user-generated content

Contact Info

Product

Resources

About