Do Distractions During Web Survey Completion Affect Data Quality? Findings From a Laboratory Experiment

Wenz, Alexander

doi:10.1177/0894439319851503

Cited by 17 publications

(20 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, research that uses further response quality indicators is required. Fifth, it would be desirable if future studies employ more refined situational factors, such as noisy or quiet environments (see Wenz, 2019), to explain total acceleration of smartphones during web survey completion. Finally, we only focused on two motion conditions (standing or walking).…”

Section: Discussionmentioning

confidence: 99%

Motion instructions in surveys: Compliance, acceleration, and response quality

Höhne

Revilla

Schlosser

2019

International Journal of Market Research

View full text Add to dashboard Cite

The increased use of smartphones in web survey responding did not only raise new research questions but also fostered new ways to research survey completion behavior. Smartphones have many built-in sensors, such as accelerometers that measure acceleration (i.e., the rate of change of velocity of an object over time). Sensor data establish new research opportunities by providing information about physical completion conditions that, for instance, can affect response quality. In this study, we explore three research questions: (1) To what extent do respondents accept to comply with motion instructions? (2) What variables affect the acceleration of smartphones? (3) Do different motion levels affect response quality? We conducted a smartphone web survey experiment using the Netquest opt-in panel in Spain and asked respondents to stand at a fix point or walk around while answering five single questions. The results reveal high compliance with motion instructions, with compliance being higher in the standing than in the walking condition. We also discovered that several variables, such as the presence of third parties, increase the acceleration of smartphones. However, the quality of responses to the five single questions did not differ significantly between the motion conditions, a finding that is in line with previous research. Our findings provide new insights into how compliance changes with motion tasks and suggest that the collection of acceleration data is a feasible and fruitful way to explore survey completion behavior. The findings also indicate that refined research on the connection between motion levels and response quality is necessary.

show abstract

Section: Discussionmentioning

confidence: 99%

Motion instructions in surveys: Compliance, acceleration, and response quality

Höhne

Revilla

Schlosser

2019

International Journal of Market Research

View full text Add to dashboard Cite

show abstract

“…While some earlier studies have confirmed indications of some of these effects, as summarised in, e.g., Couper et al (2017) and de Leeuw (2018), some recent experimental studies have somewhat surprisingly found small or no device effects, generally concluding that smartphone responses are of comparable quality to computer responses (e.g. Antoun et al 2017;Lugtig and Toepoel 2016;Schlosser and Mays 2018;Tourangeau et al 2018;Wenz 2019). The only two studies in the SP literature in environmental and resource economics comparing devices that we are aware of have generally come to the same conclusion.…”

Section: Introductionmentioning

confidence: 91%

Ponderous, Proficient or Professional? Survey Experience and Smartphone Effects in Stated Preference Research

Sandorf

Grimsrud

Lindhjem

2022

Environ Resource Econ

View full text Add to dashboard Cite

Stated preference surveys are increasingly conducted online using internet panel samples, where a fast-growing share of respondents answer on smartphones. These panel members range from novices to “professionals” in terms of previous survey experience. Little is known about the potential effects of smartphone responding and survey experience on the data quality of stated preference surveys. This paper uses a discrete choice experiment dataset on the Norwegian population’s willingness to pay to plant climate forests to explore how these two factors affect data quality. These data by type of response device, gathered using a probability-based internet panel, were combined with a unique dataset obtained from the survey company on respondents’ actual experience answering surveys on different types of devices. Our results show that differences in elicited preferences between smartphone and computer respondents are not caused by the device used, suggesting that initial concerns about smartphone responses may be exaggerated. Furthermore, more experience is associated with an increasing scale parameter (indicating lower error variance), but at a decreasing rate; and a higher propensity to choose the status quo (indicating possible simplifying strategies employed by respondents). Combined this suggest some optimal level of experience that is neither too high nor too low. We discuss the implications of our results for stated preference research and provide a few avenues for future research.

show abstract

“…Consequently, smartphone respondents are frequently surrounded by third parties during survey participation (Toninelli & Revilla, 2016b), which might have a negative impact on response behavior because respondents can get distracted due to their environment. Lynn and Kaminska (2012) distinguish three types of distractions during survey participation: (1) distractions that demand aural attention, such as music playing in the background (see also Wenz, 2019), (2) distractions that demand visual attention, such as looking after children, and (3) multitasking, such as having a conversation. Indeed, Toninelli and Revilla (2016a) show that smartphone respondents report more distractions and/or multitasking behavior, such as watching TV and talking with other people, than PC respondents.…”

Section: Introductionmentioning

confidence: 99%

“…However, one problem associated with these findings on distractions and multitasking is that they are based on self-reports, implying that respondents must admit that they might not pay constant attention. Thus, self-reports on distractions and multitasking are subject to social desirability bias and recall error, which might cause imprecise measures (Wenz, 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Previous studies on distractions and multitasking proposed to overcome the drawbacks of self-reports by passively collecting paradata 1 during web survey completion. For instance, Sendelbah et al (2016) suggest to use so-called JavaScript "OnBlur" functions that measure the in-/activity of web survey pages-whether the browser tab or window that hosts the web survey is the active or processed one (see Callegaro, 2013;Couper & Peterson, 2017;Diedenhofen & Musch, 2017;Höhne et al, 2017, 2019Revilla & Couper, 2018aSendelbah et al, 2016)-to infer on-device multitasking, such as switching browser tab or window to check incoming emails or social media notifications. show that on-device multitasking gathered by means of OnBlur functions occurs among 10% of all smartphone respondents in a web survey.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Completion Conditions and Response Behavior in Smartphone Surveys: A Prediction Approach Using Acceleration Data

Kern

Höhne

Schlosser

et al. 2020

Social Science Computer Review

View full text Add to dashboard Cite

This study utilizes acceleration data from smartphone sensors to predict motion conditions of smartphone respondents. Specifically, we predict whether respondents are moving or nonmoving on a survey page level to learn about distractions and the situational conditions under which respondents complete smartphone surveys. The predicted motion conditions allow us to (1) estimate the proportion of smartphone respondents who are moving during survey completion and (2) compare the response behavior of moving and nonmoving respondents. Our analytical strategy consists of two steps. First, we use data from a lab experiment that systematically varied motion conditions of smartphone respondents and train a prediction model that is able to accurately infer respondents’ motion conditions based on acceleration data. Second, we use the trained model to predict motion conditions of respondents in two cross-sectional surveys in order to compare response behavior of respondents with different motion conditions in a field setting. Our results indicate that active movement during survey completion is a relatively rare phenomenon, as only about 3%–4% of respondents were predicted as moving in both cross-sectional surveys. When comparing respondents based on their predicted motion conditions, we observe longer completion times of moving respondents. However, we observe little differences when comparing moving and nonmoving respondents with respect to indicators of superficial responding, indicating that moving during survey completion does not pose a severe threat to data quality.

show abstract

Do Distractions During Web Survey Completion Affect Data Quality? Findings From a Laboratory Experiment

Cited by 17 publications

References 45 publications

Motion instructions in surveys: Compliance, acceleration, and response quality

Motion instructions in surveys: Compliance, acceleration, and response quality

Ponderous, Proficient or Professional? Survey Experience and Smartphone Effects in Stated Preference Research

Completion Conditions and Response Behavior in Smartphone Surveys: A Prediction Approach Using Acceleration Data

Contact Info

Product

Resources

About