Benchmarking Crisis in Social Media Analytics: A Solution for the Data-Sharing Problem

Abstract

Computational social science uses computational and statistical methods in order to evaluate social interaction. The public availability of data sets is thus a necessary precondition for reliable and replicable research. These data allow researchers to benchmark the computational methods they develop, test the generalizability of their findings, and build confidence in their results. When social media data are concerned, data sharing is often restricted for legal or privacy reasons, which makes the comparison of methods and the replicability of research results infeasible. Social media analytics research, consequently, faces an integrity crisis. How is it possible to create trust in computational or statistical analyses, when they cannot be validated by third parties? In this work, we explore this well-known, yet little discussed, problem for social media analytics. We investigate how this problem can be solved by looking at related computational research areas. Moreover, we propose and implement a prototype to address the problem in the form of a new evaluation framework that enables the comparison of algorithms without the need to exchange data directly, while maintaining flexibility for the algorithm design.

The COVID‐19 scamdemic: A survey of phishing attacks and their countermeasures during COVID‐19

“…The first issue – that is, the limited availability of reference datasets – can be traced back to a combination of long‐known and covid‐related challenges. Firstly, building high‐quality scientific datasets have always represented a very demanding task [ 98 ]. In addition, the impact and the recency of the pandemic left even less time and resources for scholars to tackle this task.…”

“…High quality and reference datasets represent an important resource to foster research and experimentation on novel scientific issues [ 97 ]. However, building such resources is notoriously challenging and time‐demanding [ 98 ]. In the case of COVID‐19 phishing attacks, publicly available reference datasets are few and far between.…”

Promoting and countering misinformation during Australia’s 2019–2020 bushfires: a case study of polarisation

“…For these reasons, some of the content observed by Graham and Keller was expected to be missing from our dataset. This lack of consistency between social media datasets for comparative analyses is a growing challenge recently identified in the benchmarking literature (Assenmacher et al 2021 ).…”

Promoting and countering misinformation during Australia's 2019-2020 bushfires: A case study of polarisation

“…For these reasons, some of the content observed by Graham and Keller were expected to be missing from our dataset. This lack of consistency between social media datasets for comparative analyses is a growing challenge recently identified in the benchmarking literature (Assenmacher et al, 2021). A final contrast dataset was obtained via the Real-Time Analytics Platform for Interactive Data Mining (RAPID) (Lim et al, 2018) consisting of tweets containing the term '#brexit' during the same period.…”