Changing Perspectives: Is It Sufficient to Detect Social Bots?

Grimme, Christian; Assenmacher, Dennis; Clever, Lena

doi:10.1007/978-3-319-91521-0_32

Cited by 82 publications

(63 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Cresci, Petrocchi, Spognardi, and Tognazzi (2019) proposed the use of evolutionary algorithms to improve social bot skills. Grimme et al (2018) employed a hybrid approach involving automatic and manual actions to achieve bots that would be classified as human by a supervised bot detection system. Despite the good intention of pointing to weaknesses in existing systems, this research might also inspire bot creators and give them a competitive advantage.…”

Section: Bot Detection Methodsmentioning

confidence: 99%

Arming the public with artificial intelligence to counter social bots

Yang

Varol

Davis

et al. 2019

Human Behav and Emerg Tech

325

262

View full text Add to dashboard Cite

The increased relevance of social media in our daily life has been accompanied by efforts to manipulate online conversations and opinions. Deceptive social bots—automated or semi‐automated accounts designed to impersonate humans—have been successfully exploited for these kinds of abuse. Researchers have responded by developing artificial intelligence (AI) tools to arm the public in the fight against social bots. Here we review the literature on different types of bots, their impact, and detection methods. We use the case study of Botometer, a popular bot detection tool developed at Indiana University, to illustrate how people interact with AI countermeasures. A user experience survey suggests that bot detection has become an integral part of the social media experience for many users. However, barriers in interpreting the output of AI tools can lead to fundamental misunderstandings. The arms race between machine learning methods to develop sophisticated bots and effective countermeasures makes it necessary to update the training data and features of detection tools. We again use the Botometer case to illustrate both algorithmic and interpretability improvements of bot scores, designed to meet user expectations. We conclude by discussing how future AI developments may affect the fight between malicious bots and the public.

show abstract

Section: Bot Detection Methodsmentioning

confidence: 99%

Arming the public with artificial intelligence to counter social bots

Yang

Varol

Davis

et al. 2019

Human Behav and Emerg Tech

325

262

View full text Add to dashboard Cite

show abstract

“…Second, we present a theoretically informed method for the detection of a disinformation campaign. We argue that past research's predominant focus on automated accounts, famously known as "social bots", (see Howard & Kollanyi, 2016;Varol, Ferrara, Davis, Menczer, & Flammini, 2017) misses its target since reports on recent astroturfing campaigns suggest that they are often at least partially run by actual humans (so-called "cyborgs": Chu, Gianvecchio, Wang, & Jajodia, 2012), which may shield the accounts from detection strategies focused on automated behavior (Grimme, Assenmacher, & Adam, 2018). Using bot detection to study astroturfing betrays a fundamental conceptual mismatch: bots are one tool that can be used in an astroturfing campaign, but not all bots are part of it, and reversely, not all astroturfing accounts are bots.…”

Section: Introductionmentioning

confidence: 95%

Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign

Keller

Schoch

Stier

et al. 2019

Political Communication

198

120

View full text Add to dashboard Cite

Political astroturfing, a centrally coordinated disinformation campaign in which participants pretend to be ordinary citizens acting independently, has the potential to influence electoral outcomes and other forms of political behavior. Yet, it is hard to evaluate the scope and effectiveness of political astroturfing without "ground truth" information, such as the verified identity of its agents and instigators. In this paper, we study the South Korean National Information Service's (NIS) disinformation campaign during the presidential election in 2012, taking advantage of a list of campaign accounts published in court proceedings. Features that best distinguish these accounts from regular users in contemporaneously collected Twitter data are traces left by coordination among astroturfing agents, instead of the individual account characteristics typically used in related approaches such as social bot detection. We develop a methodology that exploits these distinct empirical patterns to identify additional likely astroturfing accounts and validate this detection strategy by analyzing their messages and current account status. However, an analysis relying on Twitter influence metrics shows that the known and suspect NIS accounts only had a limited impact on political social media discussions. By using the principal-agent framework to analyze one of the earliest revealed instances of political astroturfing, we improve on extant methodological approaches to detect disinformation campaigns and ground them more firmly in social science theory.

show abstract

“…RQ 3 With regard to the diagnostic ability are there differences with the performance of Botometer between languages? Grimme et al (2018) showed in their study with 3 bots in an experimental setting that the classification score is not stable over time. Therefore, we are interested in not only measuring the Botometer score once but instead tracking accounts over a longer period of time.…”

Section: Research Questionsmentioning

confidence: 99%

“…However, Botometer is not above criticism. Grimme, Assenmacher, and Adam (2018) show in their study, for example, that Botometer could not classify the hybrid and full automation bot accounts that the authors had created precisely. The original creators are also aware of the potential limitations of their tool and admit that "many machine learning algorithms, including those used by Botometer, do not make it easy to interpret their classifications" (Yang et al, 2019, p. 58).…”

Section: Introductionmentioning

confidence: 98%

The False Positive Problem of Automatic Bot Detection in Social Science Research

Rauchfleisch

Kaiser

2020

SSRN Journal

View full text Add to dashboard Cite

The identification of bots is an important and complicated task. The bot classifier Botometer was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer 's diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n=4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer 's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed to identify bots.

show abstract

Changing Perspectives: Is It Sufficient to Detect Social Bots?

Cited by 82 publications

References 13 publications

Arming the public with artificial intelligence to counter social bots

Arming the public with artificial intelligence to counter social bots

Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign

The False Positive Problem of Automatic Bot Detection in Social Science Research

Contact Info

Product

Resources

About