Estimating the Nonresponse Bias Due to Refusals in Telephone Surveys

Abstract

response rates frequently encountered in survey research threaten the external validity of findings? Because the computation of unbiased sample estimates in probability sampling requires that each element in the population have a known or calculable nonzero probability of selection (Kish, 1965), °,reat caution is usually exercised in the initial selection of a sample. Respondent refusals, however, can have an effect-of unknown magnitude-on sample estimates. One way of dealing with this problem is to try to minimize the nonresponse rate. Although there has been much experimentation with methods and technologies to minimize nonresponse, 1 even successful 1 Extensive reviews of the effectiveness of these techniques for mail surveys were published by Kanuk and Berenson (1975) and Linsky (1975). Both of these articles surveyed existing knowledge of the effectiveness of followups and reminders, preliminary notification, questionnaire length, survey sponsorship, postal options, personalizaAbstract This paper attempts to aid the process of accumulating the necessary information for making more informed judgments about the effects of nonresponse under different conditions. Two measures, which permit quantifiable nonsubjective assessment of the effects of nonresponse on sample estimates, are introduced and are used to examine the effects of respondent refusals in a random-digit-dialed general population telephone survey of over 1,200 households as the response rate is increased from 74.5 percent to 86.8 percent. By applying these measures under a wide range of conditions, the adequacy of various response rates may be assessed and more rational decisions made about the costs and benefits of devoting extraordinary resources to minimizing nonresponse.

The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms

“…Some studies documented under-representation of low education respondents and over-representation of high education respondents (e.g., Chang and Krosnick 2001;Mulry-Liggan 1983). Likewise, some researchers have found that more educated people are easier to locate for later waves of a panel survey (Schejbal and Lavrakas 1995) and less likely to be interviewed as a result of a refusal conversion (O'Neil 1979;Retzer, Schipani, and Cho 2004). However, other studies have found that more educated people require more call attempts (Merkle, Bauman, and Lavrakas 1993) and that surveys with higher response rates may over-represent high education respondents more than surveys with lower response rates (Keeter et al 2000).…”

“…Older people are also easier to locate for later waves of panel surveys (Schejbal and Lavrakas 1995), perhaps because they are more tied to the community and less likely to move between waves of 16 panel surveys. However, considerable evidence also suggests that older people may be more likely to refuse to be interviewed and may make up larger proportion of respondents who require a refusal conversion than respondents who do not (Massey, et al 1981;O'Neil 1979;Retzer, Schipani, and Cho 2004;Struebbe et al 1986). …”

“…Although this is larger than the proportion that would be expected by chance alone, the 14 differences were all small in magnitude. Other surveys have found comparably small effects of response rates on substantive responses Singer 2000, 2005;Groves, Presser, and Dipko 2004;Merkle, Bauman, and Lavrakas 1993;O'Neil 1979;Smith 1984).…”

“…One aspect of survey administration is the maximum number of times that interviewers attempt to reach each household, after which the telephone number is retired from the active sample. Higher maximum numbers of call attempts have been found to be associated with higher response rates in some studies (e.g., Massey et al 1981;Merkle, Bauman, and Lavrakas 1993;O'Neil 1979;Traugott 1987). This effect is not linear; each additional call attempt increases response rates less than the previous attempt does.…”

“…focused on substantive variables has concluded that response rates are unrelated to or only very weakly related to the distributions of substantive responses (e.g., Singer 2000, 2005;Groves, Presser, and Dipko 2004;Keeter et al 2000;Merkle, Bauman, and Lavrakas 1993;O'Neil 1979;Smith 1984). For example, comparing two similar surveys with different response rates, Keeter et al (2000) found statistically significant differences for only 14 of 91 items they compared.…”