Gamified Inoculation Interventions Do Not Improve Discrimination Between True and Fake News: Reanalyzing Existing Research With Receiver Operating Characteristic Analysis

Abstract

Gamified psychological interventions designed to improve detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (i.e., Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method unique to signal detection theory (SDT) that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, we found that Bad News and Go Viral! did not improve discrimination, but rather elicited more conservative responding (i.e., responding “true” less often). These novel findings suggest that the current gamified psychological interventions designed to improve fake news detection are not as effective as previously thought. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.

Effects of inductive learning and gamification on news veracity discernment.

“…Although more recent articles have addressed some of these problems, there is arguably still scope for improvement. Furthermore, a graphical analysis of the results of these prior investigations (Figure 1) suggests that when people are not certain about the veracity of news items to begin with, Bad News may simply make them generally more skeptical of news items, regardless of whether they are true or fake (see Modirrousta-Galian & Higham, 2022, for further discussion on this issue). Reduced belief in truth (e.g., that COVID-19 booster vaccines are important) can sometimes be more damaging than increased belief in falsehoods (e.g., that 5G towers cause COVID-19).…”

The (Un)Intended Consequences of Emphasizing the Threats of Mis- and Disinformation

“…Although the effectiveness of news media literacy interventions and pre-bunking techniques requires more empirical research, our findings suggest that it is important to look at the discernment between factually accurate information and mis-or disinformation to measure the effectiveness of interventions (see also e.g., Modirrousta-Galian & Higham, 2022). Hence, under some conditions, exposure to a media literacy intervention can lower the credibility of information irrespective of its veracity.…”

“…Thus, to comprehensively measure the effectiveness of warning labels and media literacy interventions, it is important to distinguish between the acceptance of misinformed claims and factually accurate information. Considering that experimental research found that being exposed to a warning label before seeing factually accurate information can also lower the trustworthiness of true information (Hameleers, 2022;Modirrousta-Galian & Higham, 2022), this article explores how the presentation of a warning label combined with a media literacy intervention placed before factually accurate information and mis-or disinformation influences the rating of both types of information.…”

Psychological inoculation protects against the social media infodemic

“…These effects were asymmetrical, with a larger suppression effect estimated for misinformation than for legitimate information, suggesting some improvement in ability to discriminate among inoculated participants. Future experiments should aim to further explore the mechanisms by which inoculation works, as there is recent evidence that it may catalyse more conservative engagement with posts rather than substantially improving discernment (Modirrousta-Galian & Higham, 2022). If this is indeed the case, social media platforms, policymakers and regulators will be forced to weigh the benefits of decreasing the spread of misinformation via inoculation against the social costs associated with a reduction in the dissemination of legitimate information.…”