Examining the Coherence of the Top Ranked Tweet Topics

Abstract

Topic modelling approaches help scholars to examine the topics discussed in a corpus. Due to the popularity of Twitter, two distinct methods have been proposed to accommodate the brevity of tweets: the tweet pooling method and Twitter LDA. Both of these methods demonstrate a higher performance in producing more interpretable topics than the standard Latent Dirichlet Allocation (LDA) when applied on tweets. However, while various metrics have been proposed to estimate the coherence of the generated topics from tweets, the coherence of the top ranked topics, those that are most likely to be examined by users, has not been investigated. In addition, the effect of the number of generated topics K on the topic coherence scores has not been studied. In this paper, we conduct large-scale experiments using three topic modelling approaches over two Twitter datasets, and apply a state-of-the-art coherence metric to study the coherence of the top ranked topics and how K affects such coherence. Inspired by ranking metrics such as precision at n, we use coherence at n to assess the coherence of a topic model. To verify our results, we conduct a pairwise user study to obtain human preferences over topics. Our findings are threefold: we find evidence that Twitter LDA outperforms both LDA and the tweet pooling method because the top ranked topics it generates have more coherence; we demonstrate that a larger number of topics (K) helps to generate topics with more coherence; and finally, we show that coherence at n is more effective when evaluating the coherence of a topic model than the average coherence score.

Profiling the Spatial Structure of London: From Individual Tweets to Aggregated Functional Zones

“…In general, the coherence values increased along with the number of topics as aforementioned. The result is in line with that in the data mining literature [49]. Conversely, the mean value of spatial autocorrelation decreased.…”

“…In general, the higher the coherence value, the better the quality of the topics. We implemented a grid search of best topic models and ended up with the same conclusion as that in [49]. Increasing the number of topics (T) leads to higher coherence values.…”

Votes on Twitter: Assessing Candidate Preferences and Topics of Discussion During the 2016 U.S. Presidential Election

“…We show the coherence of the topic models extracted from the two candidate communities in Appendix Figure B1. The coherence results are consistent with Fang, MacDonald, Ounis, and Habel (2016a): the average coherence of a topic model decreases when the number of topics increases; however, the increasing line of c@10/20/30 in Figure B1 indicates that the top-ranked topics in a topic model are much easier to understand as K increases. Among proClinton topic models, we found the coherence (c@10/20/30) of topics becomes stable when K reaches 70, and for proTrump, when K reaches 60.…”

Exploring Time-Sensitive Variational Bayesian Inference LDA for Social Media Data

“…Meanwhile, we also examine the top 2/7 2 most coherent topics in a model for more effective coherence evaluation, i.e. C@2 & C@7 metrics, following to [24].…”