Methods for Evaluating Respondent Attrition in Web-Based Surveys

Abstract

BackgroundElectronic surveys are convenient, cost effective, and increasingly popular tools for collecting information. While the online platform allows researchers to recruit and enroll more participants, there is an increased risk of participant dropout in Web-based research. Often, these dropout trends are simply reported, adjusted for, or ignored altogether.ObjectiveTo propose a conceptual framework that analyzes respondent attrition and demonstrates the utility of these methods with existing survey data.MethodsFirst, we suggest visualization of attrition trends using bar charts and survival curves. Next, we propose a generalized linear mixed model (GLMM) to detect or confirm significant attrition points. Finally, we suggest applications of existing statistical methods to investigate the effect of internal survey characteristics and patient characteristics on dropout. In order to apply this framework, we conducted a case study; a seventeen-item Informed Decision-Making (IDM) module addressing how and why patients make decisions about cancer screening.ResultsUsing the framework, we were able to find significant attrition points at Questions 4, 6, 7, and 9, and were also able to identify participant responses and characteristics associated with dropout at these points and overall.ConclusionsWhen these methods were applied to survey data, significant attrition trends were revealed, both visually and empirically, that can inspire researchers to investigate the factors associated with survey dropout, address whether survey completion is associated with health outcomes, and compare attrition patterns between groups. The framework can be used to extract information beyond simple responses, can be useful during survey development, and can help determine the external validity of survey results.

Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey

“…This may lead to a conceptualization of survey fatigue on a spectrum rather than as a binary outcome. The attrition approach has been previously described for Web-based surveys ( Hochheimer et al, 2016 ). More detailed analysis of step offs and straight-line answering may provide feedback for how to modify a survey to reduce the rate of fatigue.…”

“…Respondent fatigue, also known as survey fatigue, is a common problem in the collection of survey data ( Whelan, 2008 ; Ben-Nun, 2008 ). It refers to the situation in which respondents give less thoughtful answers to questions in the later parts of a survey, or prematurely terminate participation ( Whelan, 2008 ; Ben-Nun, 2008 ; Hochheimer et al, 2016 ). This may be detected when there is straight-line answering, where the respondent chooses e.g., the first option of a multiple choice survey for multiple question in a row.…”

“…It may also be present if the respondent leaves text response fields blank or if the respondent chooses the “default” response on a slider bar. Finally, fatigue may be present if the respondent fails to complete the survey ( Ben-Nun, 2008 ; Hochheimer et al, 2016 ). Factors that are known to influence respondent fatigue include survey length, survey topic, question complexity, and question type (open-ended questions tend to induce more fatigue) ( Ben-Nun, 2008 ).…”

Harnessing Information Technology to Inform Patients Facing Routine Decisions: Cancer Screening as a Test Case

“…A detailed analysis of module attrition has been previously reported. 44 When the module was used, paradata indicated that 291 patients ( Note: Given the large sample size, all differences across groups (noncompleters, completers, and nonstarters) were statistically significant (P <.001) with the exception of Medicaid insurance type.…”

Relationship Between Psychological Responses and the Appraisal of Risk Communication During the Early Phase of the COVID-19 Pandemic: A Two-Wave Study of Community Residents in China

“…Those who we were unable to retrospectively follow up fell into attrition. In many longitudinal studies, observations across waves can be missing for various reasons, and the attrition rate for web-based surveys is especially high (25). Another reason for our high attrition rate may be that we use the forced response mode; people will stop working on the survey if they are asked questions they do not wish to answer (26), although some of our questions included a "no answer" option.…”