Curbing curbstoning: Distributional methods to detect survey data fabrication by third-parties.

Abstract

Curbstoning, the willful fabrication of survey responses by outside data collectors, threatens the integrity of the inferences drawn from data. Researchers who outsource data collection to survey collection panels, field interviewers, or research assistants should validate whether each collection agent actually collected the data. Our review of the survey auditing literature demonstrates a consistent presence of curbstoning, even at professional levels. This study proposes several general simple survey questions that have statistical distributions known a priori, as a method to detect curbstoning. By exploiting common deficiencies in statistical understanding, survey collectors imputing data to these questions can leverage empirically known distributions to determine deviation from the expected distribution of responses. We examined both authentic and fabricated surveys that included these questions and we compared the observed distributions with the expected distributions. The majority of the proposed methods had Type I error rates near or below the specified alpha level (.05). The methods demonstrated the ability to detect false responses correctly 48%-90% of the time across two samples when surveying at least 50 participants. While the methods varied in effectiveness, combining these methods demonstrated the highest statistical power, with Type I error rates lower than 1%. Additionally, even in situations with smaller sample sizes (e.g., N = 30), combining these methods allows them to be effective in detecting curbstoning. These methods provide a simple and generalizable way for researchers not present during data collection to possess accurate data.

VOIS: Voice Over Internet Surveys

“…By recording a participant's voice to indicate their response, researchers are less able to realistically fabricate the response. Additionally, it serves as a check that a participant was in fact contacted, and the responses are not simply a data collector's imputation (i.e., curbstoning; for a review see Hernandez et al, 2021).…”

“…Due to the difficulty of fabricating large amounts of participants' vocal recordings, potential fabricators would be required to actually put in more work to fabricate responses if VOIS was included as a response option. Additionally, alternative proposed methods using participants' responses to innocuous questions and matching response distributions to the expected known distribution of typical, non-fabricated responses can be facilitated using VOIS (Hernandez et al, 2021). The proposed method in Hernandez et al (2021) suggests using a combination of common questions that participants can easily respond to in order to check for deviations from expected statistical distributions.…”

“…Additionally, alternative proposed methods using participants' responses to innocuous questions and matching response distributions to the expected known distribution of typical, non-fabricated responses can be facilitated using VOIS (Hernandez et al, 2021). The proposed method in Hernandez et al (2021) suggests using a combination of common questions that participants can easily respond to in order to check for deviations from expected statistical distributions. These include asking participants for non-identifying, but easy to answer questions such as their address number, birthdate, or the last four digits of their phone number.…”