Looking for Answers: Identifying Search Behavior and Improving Knowledge-Based Data Quality in Online Surveys

TRENDS: How Internet Search Undermines the Validity of Political Knowledge Measures

“…Considering these challenges, our first recommendation for researchers is to actively discourage outside search. Simple instructions to refrain from looking up answers can reduce search engine use (Motta, Callaghan, and Smith 2016; Vezzoni and Ladini 2017), but do not eliminate it, as shown in Study 2. A more effective tactic is to ask respondents to commit to not looking up answers, but even this approach is not perfect (Clifford and Jerit 2016).…”

“…As surveys are increasingly administered online, however, respondents are able to search the web for answers, potentially altering what scholars are measuring with factual knowledge questions. A recent body of research demonstrates that outside search occurs and can affect the estimated levels of political knowledge among the public (Burnett 2016; Clifford and Jerit 2014, 2016; Motta, Callaghan, and Smith 2016; Shulman and Boster 2014), but that literature has yet to establish how search behavior affects the validity of knowledge measures. Across experimental and observational studies, we find a consistent pattern of results—namely, that search engine use reduces the validity of political knowledge measures and undermines the ability to replicate canonical findings in the public opinion literature.…”

“…First, after completing the knowledge questions, respondents were asked whether they looked up any answers. Second, a difficult “catch” question was inserted in random order among the political knowledge items (Bullock et al 2015; Motta, Callaghan, and Smith 2016). The catch question asked about the year of an obscure court case ( United States v. Segui ), and respondents who provided the correct year were assumed to have looked up the answer.…”

“…The four items asked respondents to identify the job or political office held by Paul Ryan, the political party in control of the House of Representatives, the length of a U.S. Senate term, and who nominates judges to the Federal Courts (again, with no “don’t know” option). Participants were told to answer knowledge questions “to the best of your ability” and were instructed to not look up answers online (Motta, Callaghan, and Smith 2016; Vezzoni and Ladini 2017). Search engine use was measured with a difficult catch question, measured immediately after the knowledge items.…”

“…Outside search has been detected in several participant populations with search rates reaching as high as 41 percent. Evidence of search engine use comes from a variety of sources, including studies showing higher factual knowledge scores in the online condition of randomized mode experiments (Burnett 2016; Clifford and Jerit 2014), lower scores when respondents are randomly assigned instructions not to search (Clifford and Jerit 2016; Motta, Callaghan, and Smith 2016; Vezzoni and Ladini 2017), correct answers to extremely difficult and obscure open-ended questions (“catch” questions; Motta, Callaghan, and Smith 2016), and the outright admission of searching (Clifford and Jerit 2016; Jensen and Thomsen 2014). Outside search is common even in high-quality surveys that have explicitly instructed respondents not to look up answers.…”