A Practical Guide to Doing Behavioral Research on Fake News and Misinformation

Abstract

Coincident with the global rise in concern about the spread of misinformation on social media, there has been influx of behavioral research on so-called “fake news” (fabricated or false news headlines that are presented as if legitimate) and other forms of misinformation. These studies often present participants with news content that varies on relevant dimensions (e.g., true v. false, politically consistent v. inconsistent, etc.) and ask participants to make judgments (e.g., accuracy) or choices (e.g., whether they would share it on social media). This guide is intended to help researchers navigate the unique challenges that come with this type of research. Principle among these issues is that the nature of news content that is being spread on social media (whether it is false, misleading, or true) is a moving target that reflects current affairs in the context of interest. Steps are required if one wishes to present stimuli that allow generalization from the study to the real-world phenomenon of online misinformation. Furthermore, the selection of content to include can be highly consequential for the study’s outcome, and researcher biases can easily result in biases in a stimulus set. As such, we advocate for pretesting materials and, to this end, report our own pretest of 224 recent true and false news headlines, both relating to U.S. political issues and the COVID-19 pandemic. These headlines may be of use in the short term, but, more importantly, the pretest is intended to serve as an example of best practices in a quickly evolving area of research.

Psychological inoculation improves resilience against misinformation on social media

“…Study 6 had two goals: (i) to replicate the findings from study 1 (the emotional language video) 1 year after it was originally conducted and (ii) to check whether manipulating the order in which participants respond to the outcome measures for each of the stimuli (technique recognition, trustworthiness, and sharing) influences the results. This is important because eliciting (for example) the manipulativeness and/or trustworthiness of a survey item before willingness to share might influence the responses participants give for the sharing measure, as participants may be primed to think about the item’s manipulativeness before providing their sharing intentions ( 29 , 30 ). Therefore, alongside the experimental condition, participants were randomly assigned to one of three response orders [manipulativeness – trustworthiness – sharing (MTS), n = 364; trustworthiness – sharing – manipulativeness (TSM), n = 361; or sharing – manipulativeness – trustworthiness (SMT); n = 343]; see Materials and Methods for more details.…”

“…For study 6, we varied the outcome measure response order across participants, as one of the study’s goals was to check whether the order in which participants respond to the measures beneath each of the stimuli has any bearing on the results ( 29 , 30 ). Participants were randomly assigned to one of three response orders (after being randomly assigned to an experimental condition): MTS ( n = 364), TSM ( n = 361), or SMT ( n = 343).…”

“…1) This post is manipulative/is incoherent/contains a false dichotomy/ constitutes scapegoating/contains an ad hominem 2) I am confident in my assessment of this post's manipulativeness/ incoherence/of whether it contains a false dichotomy/whether it constitutes scapegoating/whether it contains an ad hominem 3) This post is trustworthy 4) I would share this post with people in my network For study 6, we varied the outcome measure response order across participants, as one of the study's goals was to check whether the order in which participants respond to the measures beneath each of the stimuli has any bearing on the results (29,30). Participants were randomly assigned to one of three response orders (after being randomly assigned to an experimental condition): MTS (n = 364), TSM (n = 361), or SMT (n = 343).…”

Story stimuli for instantiating true and false beliefs about the world

“…Researchers could also modify presentations of the vignette content to offer them in different modalities or information environments (Corneille et al, 2020 ; Fazio, Dolan, & Marsh, 2015b ). For example, with growing interest in people’s uptake of information online, the vignettes could be adapted as short-form videos (e.g., Butler et al, 2009 ), as posts on social media (e.g., Pennycook et al, 2021a ), or stripped of a narrative component completely (e.g., Fazio, Dolan, & Marsh, 2015b ; Salovich et al, 2022 ). These kinds of manipulations are possible with the current materials as they are intentionally brief, simple, flexible, allow for participant accountability, and are realistic.…”

Field Experiments on Social Media

“…Furthermore, the stimuli used in survey experiments are typically collected, or even created, by the researchers. Thus, these posts may differ in important ways from the content that users would actually experience on their own social-media newsfeeds (Pennycook, Binnendyk, et al, 2021). For example, posts in survey experiments are typically presented without the social context that is such a core feature of social media (e.g., information about which user shared the content and which other users liked or commented on the content).…”