Assessing the Effect of Data Collection Mode on Measurement

Abstract

Standard-Nutzungsbedingungen:Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden.Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen.Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in Non-technical summaryRising administration costs and falling response rates mean that many surveys that would previously have been carried out in one preferred mode of data collection are having to consider the use of mixed modes. For example, increasing numbers of surveys use a mix of modes, starting with a cheaper mode (such as telephone interviewing) which typically produces lower response rates, and following up non-respondents with face-to-face interviews. In order to decide about suitable data collection designs, survey practitioners must assess the trade-off between the potential advantages (for example in terms of financial costs and response rates) and disadvantages (for example in terms of data comparability) of mixing modes.We discuss some of the challenges in evaluating the effects of using mixed modes on measurement and hence data comparability. The main argument is that it is very difficult to provide the information survey practitioners would need, about whether and to what extent using mixed modes would affect substantive conclusions. We briefly review theories about why different modes can lead to differences in survey responses. We then discuss the methods typically used to assess mode effects on measurement and then focus on some of the challenges. These include 1) the need to avoid confounding effects and what kinds of mode effects are actually identified, 2) the sensitivity of conclusions about the existence of mode effects to statistical methods used for the analysis of experimental mode comparison data, 3) the difficulty of assessing whether measurement differences matter in practice, and 4) the assessment of which mode provides better measurement. The main focus of the paper is on analysis methods. The points raised for discussion here arose in the context of the European Social Survey (ESS), which is conducting a programme of experimental research to inform the decision about whether to allow telephone interviewing in addition to face-to-face in its future rounds. We use some examples from the ESS experiments to illustrate how we tried to deal with these issues and to stimulate discussion. The paper concludes with an outlook of how the findings from the experimental studies are informing the decision process about whether or not to mix modes of data collection on the ESS and with general implications for mixed modes research. Assessing the Effect of Data Collection Mode on MeasurementAnnette Jäckle, Car...

Changes in workplace tasks in Germany—evaluating skill and task measures

“…13 To the best of our knowledge, there is no study on effects of changing data collection modes on task composition. From the literature on method changes in general (Roberts 2007;Jäckle et al 2008), we do not expect relevant differences in the answering behaviour (measurement error); job tasks as surveyed here are not sensitive questions and thus should not provoke social desirability. All five rounds are probability samples of the German active labor force and we use comparable samples of respondents, i.e.…”

Are there differences in responses to social identity questions in face-to-face versus telephone interviews? Results of an experiment on a longitudinal survey

“…Only hearing the question and not being able to see the interviewer during a telephone interview may also make the respondent more fatigued. This may result in more off the cuff responses (satisficing) in telephone interviews (Holbrook et al, 2003), though Jäckle et al (2010) found no evidence of satisficing in their study of modes in the European Social Survey. In addition, a person is more likely to be multi-tasking during a telephone interview when the interviewer cannot see them.…”

“…There is now a wealth of research investigating the size and causes of these effects (de Leeuw, 2005;Dillman, 2000;Dillman & Tarnai, 1988;Holbrook et al, 2003;Jäckle et al, 2010;Kreuter, Presser, & Tourangeau, 2008;Lynn, 1998;Tourangeau & Smith, 1996). Overall, existing reviews comparing face-to-face and telephone (de Leeuw, 2005;de Leeuw & van der Zouwen, 1988) indicate that there appear to be relatively few differences between the two interviewer-administered modes of telephone and face-to-face.…”

“…These include differences in sampling frames, response rates, survey organisation, questionnaire design and format (Jäckle et al, 2010;Roberts, 2007) as well as observed and unobserved differences between the respondents to the different modes. If such factors can be equalised across modes by a consistent survey context (the same sample and survey organisation), by random allocation to mode, and equivalent engagement of participants, then we expect (1) mode effects to be small, and (2) these mode effects would be attributable to the physical presence or absence of the interviewer only.…”

“…de Leeuw, 2005;Dillman, 2000;Dillman & Tarnai, 1988;Holbrook, Green, & Krosnick, 2003;Jäckle, Roberts, & Lynn, 2010;Lynn, 1998). As well as identifying those questions that tend to be most sensitive to mode effects, such as sensitive behavioural questions or attitudinal questions, the literature has also indicated ways in which the impact of mode can be mitigated by effective design (Dillman, 2000).…”