Evaluation Measures for Event Detection Techniques on Twitter Data Streams

Abstract

Abstract. Twitter's popularity as a source of up-to-date news and information is constantly increasing. In response to this trend, numerous event detection techniques have been proposed to cope with the rate and volume of social media data streams. Although most of these works conduct some evaluation of the proposed technique, a comparative study is often omitted. In this paper, we present a series of measures that we designed to support the quantitative and qualitative comparison of event detection techniques. In order to demonstrate the effectiveness of these measures, we apply them to state-of-the-art event detection techniques as well as baseline approaches using real-world Twitter streaming data.

Editorial: Survey and Experimental Analysis of Event Detection Techniques for Twitter

“…As important articles, we define search results that are from one of the top 15 news websites such as CNN, CBSNews, USAToday, BBC and Reuters. For the second part of our precision measure, we query the archive of the New York Times 6 with the five event terms as well as the specific date range. Since the number of hits (h), which are in the range between 0 and 10 both for Google (h G ) or New York Times (h NYT ), is an indicator of how important a reported event is, we calculate the final precision score for all results (N) by weighting the single results as…”

“…This article is an extended presentation of Weiler et al [6]. In comparison to the conference version, it features two additional contributions.…”

Stability Evaluation of Event Detection Techniques for Twitter

“…In this context, it would be interesting to define measures, which can be used to rank the degree of reproducibility of existing and future research work in the area of event detection. For this purpose, we created a survey [24] about the techniques and evaluations of 42 related works. With this list we can, for example, rank the works based on the availability of source code, pseudo code, or at least a very precise description of the algorithm.…”

“…We evaluate the impact of changes to the pre-processing phase by starting from the parameter settings used in our previous evaluations [26,27]. In the first experiment, we remove the pre-processing operator that suppresses retweets in the input (first operator with a dashed frame in Fig.…”

“…Currently, many evaluations of event detection appear to be non-reproducible. Weiler [24] lists the evaluation methods of a collection of 42 research works on event detection. Half of these evaluations is based on case or user studies.…”