Towards Reproducible Research of Event Detection Techniques for Twitter

Abstract

A major challenge in many research areas is reproducibility of implementations, experiments, or evaluations. New data sources and research directions complicate the reproducibility even more. For example, Twitter continues to gain popularity as a source of up-to-date news and information. As a result, numerous event detection techniques have been proposed to cope with the steadily increasing rate and volume of social media data streams. Although some of these works provide their implementation or conduct an evaluation of the proposed technique, it is almost impossible to reproduce their experiments. The main drawback is that Twitter prohibits the release of crawled datasets that are used by researchers in their experiments. In this work, we present a survey of the vast landscape of implementations, experiments, and evaluations presented by the different research works. Furthermore, we propose a reproducibility toolkit including Twistor (Twitter Stream Simulator), which can be used to simulate an artificial Twitter data stream (including events) as input for the experiments or evaluations of event detection techniques. We further present the experimental application of the reproducibility toolkit to stateof-the-art event detection techniques.

The myth of reproducibility: A review of event tracking evaluations on Twitter

“…Following Weiler et al ( 2019 ), in Sections 5, 6 we split reproducibility into four factors: the data, the ground truth, the metrics, and the algorithms. As part of our analysis on the data's reproducibility, we noted the number and sizes of the datasets used in each publication.…”

“…Event tracking literature knows well the challenges of evaluating its algorithms. Weiler et al ( 2017 ), whose body of work (Weiler et al, 2015a , b , 2016 , 2019 ) gives a broad overview of the problems, describe the evaluation process itself as “a challenging research question” independent of the event tracking one. Nevertheless, for a problem as essential as measuring progress, we still understand event tracking evaluations poorly.…”

“…Evidently, we know that event tracking desperately needs a reproducible evaluation methodology (Weiler et al, 2019 ). We also know that manual evaluations, the very antithesis of reproducibility, abound; in Weiler et al ( 2017 )'s review, 18 of 42 publications evaluated manually.…”

“…In the rest of this section, we discuss these struggles. We develop and extend (Weiler et al, 2019 )'s previous work to understand what deprives event tracking literature of reproducible data, ground truths, metrics and algorithms.…”

“…Apparently-unaware of how tweet datasets self-sanitize over time, Weiler et al ( 2019 ) proposed an alternative, the artificial stream. The artificial stream replaces the traditional corpus with a statistical distribution of background topics, formed by words, into which the researcher injects events, themselves formed by event-related keywords.…”