A practical guide to doing behavioural research on fake news and misinformation

Pennycook, Gordon; Binnendyk, Jabin; Newton, Christie; Rand, David G.

doi:10.31234/osf.io/g69ha

Cited by 16 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In each study, only participants who indicated that they use social media were allowed to participants. Participants were presented one at a time (but in a random order) a set of actual true and false news headlines taken from social media (in the format of a Facebook post; see Pennycook, Binnendyk, et al (2020) for a detailed explanation of the methodology behind headline selection). All of the false headlines were found using popular fact-checking sites such as snopes.com and factcheck.org and all of the true headlines came from reputable mainstream news sources.…”

Section: Experimental Designsmentioning

confidence: 99%

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Pennycook¹,

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Simply failing to consider accuracy when deciding what to share on social media has been shown to play an important role in the spread of online misinformation. Interventions that shift users’ attention towards the concept of accuracy – accuracy prompts or nudges – are therefore a promising approach to improve the quality of news content that users share and therefore reduce misinformation online. Here we test the replicability and generalizability of this effect by conducting a personal meta-analysis of 20 accuracy prompt survey experiments (total N=26,245) that our group ran using American participants between 2017 and 2020. These experiments used a wide range of different accuracy prompts tested using a large variety of headline sets and with participants recruited from qualitatively different subject pools. We find that overall, accuracy prompts increased sharing discernment (difference in sharing intentions for true relative to false headlines) by 72% relative to the control, and that this effect was primarily driven by reducing sharing intentions for false headlines (10% reduction relative to the control). The magnitude of the accuracy prompt effect on sharing discernment did not significantly differ for headlines about politics versus COVID-19, and was larger for headline sets where users were less likely to distinguish between true and false headlines at baseline. With respect to individual-level variables, the treatment effect on sharing discernment was not significantly moderated by gender, race, or political ideology, and was significantly larger for older participants, participants who were higher on cognitive reflection, and participants who passed more attention check questions. These results suggest that accuracy prompt effects are replicable and generalize across prompts and headlines, and thus offer a promising approach for fighting against misinformation.

show abstract

Section: Experimental Designsmentioning

confidence: 99%

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Pennycook¹,

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Following classic (Clark & Watson, 1995;Loevinger, 1957) and recent (Boateng et al, 2018;Rosellini & Brown, 2021;Zickar, 2020) psychometrics guidelines, and taking into account insights from misinformation scholars (Pennycook, Binnendyk, et al, 2020;Roozenbeek, Maertens, et al, 2021), we devised a four-stage scale development protocol (i.e., 1-item generation, 2-expert filtering, 3-quality control, and 4-data-driven selection), shown in Figure 1.…”

Section: Propertiesmentioning

confidence: 99%

“…There is a debate in the literature on whether the misinformation items administered in misinformation studies should be actual news items circulating in society, or news items created by experts that are fictional but feature common misinformation techniques. The former approach arguably provides better ecological validity (Pennycook, Binnendyk, et al, 2020), while the latter provides a cleaner and less confounded measure since it is less influenced by memory and identity effects (van der Linden & . Considering these two approaches and reflecting on representative stimulus sampling (Dhami et al, 2004), we opted for a novel approach that combines the best of both worlds.…”

Section: Preparatory Stepsmentioning

confidence: 99%

The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment

Maertens¹,

Fm²,

Schneider³

et al. 2021

Preprint

View full text Add to dashboard Cite

Interest in the psychology of misinformation has exploded in recent years. Despite ample research, to date there is no psychometrically validated instrument to measure people’s ability to detect misinformation. To overcome this, we introduce the Verification done framework, an overarching interpretation schema, that simultaneously considers overall veracity discernment, and the distinct, measurable abilities (real news detection, fake news detection), and biases (distrust—negative judgement bias; naïvité—positive judgement bias) that it is composed of—thus offering a holistic and nuanced assessment. We then conduct three studies with six independent samples (Ntotal = 7,291) to develop, validate, and apply the Misinformation Susceptibility Test (MIST), the first psychometrically validated measurement instrument of veracity discernment ability. In Study 1 (N = 409), we use a neural network language model to generate items for our test, and use factor analysis and item-response theory to create the MIST-20 (20 items; <2 minutes) and MIST-8 (8 items; <1 minute). In Study 2 (N = 6,461), we confirm model fit in four representative samples (US, UK), from three different sampling platforms—Respondi, CloudResearch, and Prolific. We also explore the MIST’s nomological net, which demonstrates good convergent and discriminant validity, and generate age-, region-, and country-specific norm tables. In Study 3 (N = 421), we demonstrate how the MIST can be used in practice to test the effectiveness of interventions using the Verification done framework. We provide instructions for researchers and practitioners on how to implement the MIST as a screening tool, covariate, or framework for evaluating effects.

show abstract

“…Uncorrected contrasts also suggested that such increase was significantly stronger the validity of the content which they were seeing, which could in turn influence any subsequent sharing intention [30,32,75].…”

Section: Supporting Informationmentioning

confidence: 99%

Lateral reading and monetary incentives to sort out scientific (dis)information

Panizza¹,

Ronzani²,

Mattavelli³

et al. 2021

Preprint

View full text Add to dashboard Cite

Scientific disinformation can impose enormous economic and public health burdens. Several types of interventions have been proposed to prevent the proliferation of false information online, where most of the spreading takes place. A recently proposed strategy to help online users recognise false content is to follow the techniques of professional fact checkers, such as looking for information on other websites (lateral reading) and looking beyond the first results suggested by search engines (click restraint). In two preregistered online experiments (N = 5387), we simulated a social-media environment and set-out two interventions, one in the form of a pop-up meant to induce participants to follow such techniques, the other based on monetary incentive. In Experiment 1, we compared these interventions to a control condition. In Experiment 2 another condition was added to test the joint impact of the pop-up and the monetary incentive. We measured participants' ability to identify whether presented scientific information was scientifically (in)valid. Results revealed that while monetary incentives were overall more effective in increasing accuracy, the pop-up contributed when the post originated from an unknown source (and participants could rely less on prior information). Additional analysis on participants’ search style based on both self-report responses and objectively measured behaviour revealed that the pop-up increased the use of fact-checking strategies, and that these in turn increased accuracy. Study 2 also clarified that the pop-up and the incentive did not interfere with each other, but rather acted complementarily, suggesting that attention and literacy interventions can be designed in synergy.

show abstract

A practical guide to doing behavioural research on fake news and misinformation

Cited by 16 publications

References 20 publications

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment

Lateral reading and monetary incentives to sort out scientific (dis)information

Contact Info

Product

Resources

About