TRENDS: How Internet Search Undermines the Validity of Political Knowledge Measures

Abstract

Political knowledge is central to understanding citizens’ engagement with politics. Yet, as surveys are increasingly conducted online, participants’ ability to search the web may undermine the validity of factual knowledge measures. Recent research shows this search behavior is common, even when respondents are instructed otherwise. However, we know little about how outside search affects the validity of political knowledge measures. Using a series of experimental and observational studies, we provide consistent evidence that outside search degrades the validity of political knowledge measures. Our findings imply that researchers conducting online surveys need to take steps to discourage and diagnose search engine use.

Digital literacy and online political behavior

“…Not only might the effect of knowledge vary across levels of digital literacy, but knowledge itself might function differently for people aware that answers will always be at their fingertips. Smith et al (2020) suggest that "Scholars who are interested in measuring political knowledge should take efforts to minimize and diagnose search behavior," but this assumes a static, crystallized view of knowledge that may no longer hold for people who effectively externalize their store of factual information. This novel, transactive view of political knowledge is distinct from the traditional, static view-but overall, given the evidence in Kleinberg and Lau (2019), the former may now be more relevant for understanding how citizens engage in the process of transforming political knowledge into political preferences and actions.…”

“…These are online information-related tasks of interest to political scientists; to our knowledge, this is the first internet-based survey implementation of these types of tasks. Internet-based surveys can be a drawback for scholars studying political knowledge because they enable the possibility that respondents will "cheat" and look up the answers online (Clifford and Jerit, 2016;Smith et al, 2020). We use this feature to our advantage, explicitly instructing respondents to look up the answers and intentionally asking questions which are sufficiently obscure to US-based respondents that few could be expected to know the answer.…”

Digital Literacy and Online Political Behavior

“…Not only might the effect of knowledge vary across levels of digital literacy, but knowledge itself might function differently for people aware that answers will always be at their fingertips. Smith, Clifford, and Jerit (2020) suggest that "Scholars who are interested in measuring political knowledge should take efforts to minimize and diagnose search behavior," but this assumes a static, crystallized view of knowledge that may no longer hold for people who effectively externalize their store of factual information.…”

“…Internet-based surveys can be a drawback for scholars studying political knowledge because they enable the possibility that respondents will "cheat" and look up the answers online (Clifford and Jerit 2016;Smith, Clifford, and Jerit 2020). We use this feature to our advantage, explicitly instructing respondents to look up the answers and intentionally asking questions which are sufficiently obscure to U.S.-based respondents that few could be expected to know the answer.…”

What is the best proxy for political knowledge in surveys?

“…As Smith, Clifford, and Jerit (2020) demonstrate, fact recall is a product of political interest, whereas ability to find information is related to the motivation to do well in the test. Consequently, they conclude that ‘search engine use reduces the validity of political knowledge measures and undermines the ability to replicate canonical findings in the public opinion literature’ [ 22 ].…”