Stability Evaluation of Event Detection Techniques for Twitter

Abstract

Abstract. Twitter continues to gain popularity as a source of up-todate news and information. As a result, numerous event detection techniques have been proposed to cope with the steadily increasing rate and volume of social media data streams. Although most of these works conduct some evaluation of the proposed technique, comparing their effectiveness is a challenging task. In this paper, we examine the challenges to reproducing evaluation results for event detection techniques. We apply several event detection techniques and vary four parameters, namely time window (15 vs. 30 vs. 60 mins), stopwords (include vs. exclude), retweets (include vs. exclude), and the number of terms that define an event (1...5 terms). Our experiments use real-world Twitter streaming data and show that varying these parameters alone significantly influences the outcomes of the event detection techniques, sometimes in unforeseen ways. We conclude that even minor variations in event detection techniques may lead to major difficulties in reproducing experiments.

The myth of reproducibility: A review of event tracking evaluations on Twitter

“…Event tracking literature knows well the challenges of evaluating its algorithms. Weiler et al ( 2017 ), whose body of work (Weiler et al, 2015a , b , 2016 , 2019 ) gives a broad overview of the problems, describe the evaluation process itself as “a challenging research question” independent of the event tracking one. Nevertheless, for a problem as essential as measuring progress, we still understand event tracking evaluations poorly.…”

“…Manual efforts reassert themselves, this time to deter the prospects of implementing someone else's solution (McMinn et al, 2013 ). No one knows the nuanced design of an algorithm better than its author, and innocuous modifications can drastically affect results (Weiler et al, 2016 ; Raff, 2019 ). Thus, the simplest of techniques become pretend-baselines to establish a false state-of-the-art.…”

Towards Reproducible Research of Event Detection Techniques for Twitter

“…This choice is just one example that demonstrates the challenging and complex problem of evaluating event detection techniques automatically. As a consequence, proposals [32], [34], [35] for semi-automatic evaluation have been presented. In the following, we present the two major issues of reproducibility of event detection techniques: (i) the difficulty of reproducing the implementations of the techniques itself and (ii) the challenge to reproduce the evaluation or experiments of previous research works.…”

“…However, even if the source code is available, the diversity of implementations makes it very difficult to recreate the original environment and to exactly reproduce the research results obtained in the original work. In this context, Weiler et al [32] show that minor modifications in the different phases or parameters of event detection techniques can strongly impact the stability of their results. Another challenge is that experiments and evaluations are done in very different ways.…”