How to Extract Relevant Knowledge from Tweets?

Abstract

Abstract. Tweets exchanged over the Internet are an important source of information even if their characteristics make them difficult to analyze (e.g., a maximum of 140 characters; noisy data). In this paper, we investigate two different problems. The first one is related to the extraction of representative terms from a set of tweets. More precisely we address the following question: are traditional information retrieval measures appropriate when dealing with tweets?. The second problem is related to the evolution of tweets over time for a set of users. With the development of data mining approaches, lots of very efficient methods have been defined to extract patterns hidden in the huge amount of data available. More recently new spatio-temporal data mining approaches have specifically been defined for dealing with the huge amount of moving object data that can be obtained from the improvement in positioning technology. Due to particularity of tweets, the second question we investigate is the following: are spatio-temporal mining algorithms appropriate for better understanding the behavior of communities over time? These two problems are illustrated through real applications concerning both health and political tweets.

Exploring Spanish health social media for detecting drug effects

“…It should be noted that social media texts pose new challenges that are not present in the processing of medical literature. These new problems are the management of metadata included in the text [ 18 ], the detection of misspellings, word shortenings [ 19 , 20 ], slang and emoticons and to cope with ungrammatical phrases, among others. Moreover, while many terms present in clinical records and medical literature could be linked to domain resources, lay terms are not usually recorded in any structured resource.…”

“…Another important advantage is that such texts can be easily linked to biomedical ontologies by matching detected entities to concepts in these semantic resources. Meanwhile social media texts are markedly different from medical literature, and thereby the processing of social media texts poses additional challenges such as the management of metadata associated to the text (such as tags in tweets) [ 18 ], the detection of typos and unconventional spelling, word shortenings [ 19 , 20 ], slang and emoticons [ 21 ] and lack of punctuation marks, among others. Moreover, these texts are often very short and with an informal nature, making the processing task extremely challenging.…”

Extracting drug indications and adverse drug reactions from Spanish health social media

“…Social media texts pose additional challenges to those associated with the processing of clinical records and medical literature. These new challenges include the management of metainformation included in the text (for example as tags in tweets) (Bouillot et al, 2013), the detection of typos and unconventional spelling, word short-Figure 2: An example of the output of the system using the database. enings (Neunerdt et al, 2013;Moreira et al, 2013) and slang and emoticons (Balahur, 2013), among others.…”

Toward Optimal Feature Selection in Naive Bayes for Text Categorization

“…TF-IDF measure considers both term frequency and inverse document frequency to calculate the importance of features [25]. More recently, a generalized TF-IDF measure [26] is proposed by considering different level of hierarchies among words to effectively analyze tweet user behaviors.…”