The Effect of CATI Questions, Respondents, and Interviewers on Response Time

Abstract

In this paper, we evaluate the joint effects of question, respondent, and interviewer characteristics on response time in a telephone survey. We include question features traditionally examined, such as the length of the question and format of response options, and features that have yet to be examined that are related to the layout and format of intervieweradministered questions. We examine how these question features affect the time to ask and answer survey questions and how different interviewers vary in their administration of these questions. This paper uses paradata from the Work and Leisure Today survey and uses cross-classified random effects models. Overall, most of the variation in response time is due to question characteristics, rather than respondent or interviewer attributes. Additionally, we find that question characteristics related to necessary survey design features and respondent confusion are the primary predictors of response time, with little effect of visual design features of the question.We also find modest differences in the effects of question characteristics by interviewer experience.

The Effects of Respondent and Question Characteristics on Respondent Answering Behaviors in Telephone Interviews

“…Comprehension difficulties may arise when respondents are asked to perform complex tasks or understand challenging vocabulary. The Flesch-Kincaid reading level is a commonly used measure of question comprehension difficulty and is associated with respondent behaviors indicating comprehension and mapping difficulties and with data quality problems (Holbrook et al 2006(Holbrook et al , 2016Velez and Ashworth 2007;Lenzner 2012Lenzner , 2014Olson and Smyth 2015). We expect that questions with higher reading levels will inhibit comprehension and, thus, be associated with more requests for clarification, fewer adequate answers, and higher rates of problematic substantive and nonsubstantive (especially "don't know") answers.…”

“…If the question contains unknown terms, respondents are more likely to ask for clarification or respond with a "don't know" response, although the effects of unknown terms on other respondent behaviors are mixed (Morton-Williams and Sykes 1984;Fowler 1992;Fowler and Cannell 1996;Johnson, O'Rourke, Chavez, Sudman, Warnecke, et al 1996;Knauper et al 1997;Holbrook et al 2006;Olson and Smyth 2015). Adding definitions, either as part of the question text or as an optional statement read to the respondents, is one approach to dealing with unknown or technical terms.…”

“…Adding definitions, either as part of the question text or as an optional statement read to the respondents, is one approach to dealing with unknown or technical terms. Simple definitions allow respondents time to think and may improve comprehension and the chances of ultimately providing adequate answers, although unclear or surprising definitions are more difficult to understand and result in "don't know" responses (Conrad and Schober 2000;Tourangeau, Conrad, and Couper 2013) and longer response times (Olson and Smyth 2015). Therefore, we expect that unknown terms and definitions will increase the rate of clarification requests and the rate of problematic substantive and nonsubstantive answers and decrease adequate substantive answers.…”

“…Behavioral and attitudinal questions may require any of the retrieval types, depending on whether the question is unclear, has a long time frame, requires detailed information, requires respondents to construct an answer on the spot, or reconcile competing information on a particular domain (Tourangeau et al 2000). As such, both behavioral and attitudinal questions are answered more slowly than demographic questions (Bassili and Fletcher 1991;Yan and Tourangeau 2008;Olson and Smyth 2015). The extra effort required for either attitudinal or behavioral questions should result in increased rates of problematic substantive and nonsubstantive answers relative to demographic questions (Fowler and Cannell 1996;Dykema et al 1997;Ongena and Dijkstra 2007).…”

Examining Changes of Interview Length over the Course of the Field Period

“…Older and less educated respondents will have longer interviews due to less working memory capacity (Yan and Tourangeau 2008;Couper and Kreuter 2013;Olson and Smyth 2015). Competing time demands or distractions that result from family or work circumstances could also shorten interview length (Couper and Kreuter 2013).…”