Improving tweet stream classification by detecting changes in word probability

Abstract

We propose a classification model of tweet streams in Twitter, which are representative of document streams whose statistical properties will change over time. Our model solves several problems that hinder the classification of tweets; in particular, the problem that the probabilities of word occurrence change at different rates for different words. Our model switches between two probability estimates based on full and recent data for each word when detecting changes in word probability. This switching enables our model to achieve both accurate learning of stationary words and quick response to bursty words. We then explain how to implement our model by using a word suffix array, which is a full-text search index. Using the word suffix array allows our model to handle the temporal attributes of word n-grams effectively. Experiments on three tweet data sets demonstrate that our model offers statistically significant higher topicclassification accuracy than conventional temporally-aware classification models.

We can learn your #hashtags: Connecting tweets to explicit topics

“…Besides traditional text classification techniques, some recent works have focused on short text classification [24], [25]. [25] has extracted eight features for 5-class classification (i.e., news, events, opinions, deals, and private messages).…”

“…Limited by its small parameter space, this method cannot handle classification problem with millions of hashtags as class labels. [24] has further considered changes in word probability to classify Tweet stream, which is only based on texts. It does not consider Twitter-specific features nor user's preferences.…”

Event identification for local areas using social media streaming data

“…In the same year Alvanaki et al [1] proposed a system "enBlogue", which analyzes statistics about tags and tag pairs for identifying unusual shifts in correlations. Further recent work proposed by Nishida et al [15] shows a classification model of tweet streams for identifying changes in statistical properties on word basis, which is used for topic classification. Also in the same year Zimmermann et al [23] propose a text stream clustering method that detects, tracks and updates large and small bursts of news in a two-level topic hierarchy.…”

Classifying microblogs for disasters

“…For example, opinions are not the focus of our work. Nishida et al [2012] presented a wide range of tweet classification frameworks using a temporally aware Naïve Bayes classifier. Their experiments were conducted on a data set in which classes were defined based on their hashtags.…”