Assessing the Mechanisms of Misreporting to Filter Questions in Surveys

Abstract

To avoid asking respondents questions that do not apply to them, surveys often use filter questions that determine routing into follow-up items. Filter questions can be asked in an interleafed format, in which follow-up questions are asked immediately after each relevant filter, or in a grouped format, in which follow-up questions are asked only after multiple filters have been administered. Most previous investigations of filter questions have found that the grouped format collects more affirmative answers than the interleafed format. This result has been digitalcommons.unl.edu E c k m a n E t a l . i n P u b l i c O P i n i O n Q u a r t e r l y ( 2 0 1 4 ) 2 taken to mean that respondents in the interleafed format learn to shorten the questionnaire by answering the filter questions negatively. However, this is only one mechanism that could produce the observed differences between the two formats. Acquiescence, the tendency to answer yes to yes/no questions, could also explain the results. We conducted a telephone survey that linked filter question responses to high-quality administrative data to test two hypotheses about the mechanism of the format effect. We find strong support for motivated underreporting and less support for the acquiescence hypothesis. This is the first clear evidence that the grouped format results in more accurate answers to filter questions. However, we also find that the underreporting phenomenon does not always occur. These findings are relevant to all surveys that use multiple filter questions.

Interviewer Effects on a Network-Size Filter Question

“…Several projects have been launched in order to investigate the processes leading to such underreporting (Tourangeau et al 2015;Eckman et al 2014). Most of the results published so far refer to respondent effects.…”

The Role of Surveys in the Era of “Big Data”

“…As the reader can see, the question wording is stretching the limit of the survey tool by asking respondents to remember things with a level of precision that the human memory (in an interview setting) is not very well suited for (see also Eckman et al 2014 for a discussion on this question wording). In fact, this specific question was an object of a redesign as the committee in charge described it: "This questionnaire structure creates […] cognitive challenges" (italics added) (Dillman and House 2013, 84).…”

Total Survey Error for Longitudinal Surveys

“…This is a form of panel conditioning. For example, there is evidence that respondents remember and recognize filter questions and on purpose underreport to avoid follow-up questions [21]. This can lead to underestimation of change, if respondents avoid mentioning changes in their life in order to avoid additional questions.…”

“…As outlined earlier, one of the unique features of longitudinal surveys is that the respondent experience of taking part previously is likely to have a major impact on the decision to take part again. Thus, interview content that is interesting, engaging, and salient to respondents 21 may reduce nonresponse error, whereas the content that is in principle most relevant to the study objectives may not necessarily be so interesting, engaging or salient.…”