Assuring the quality of survey data: Incentives, detection and documentation of deviant behavior

Winker, Peter

doi:10.3233/sji-161012

Cited by 8 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, researchers should incentivize following the survey's protocol, including behaviors indicative of conscientiousness. Winker (2016) highlights that external incentives can counteract curbstoning, when individual differences associated with curbstoning are unknown during hiring.…”

Section: Educating Interviewersmentioning

confidence: 99%

Curbing curbstoning: Distributional methods to detect survey data fabrication by third-parties.

Hernandez¹,

Ristow²,

Hauenstein³

2022

Psychological Methods

View full text Add to dashboard Cite

Curbstoning, the willful fabrication of survey responses by outside data collectors, threatens the integrity of the inferences drawn from data. Researchers who outsource data collection to survey collection panels, field interviewers, or research assistants should validate whether each collection agent actually collected the data. Our review of the survey auditing literature demonstrates a consistent presence of curbstoning, even at professional levels. This study proposes several general simple survey questions that have statistical distributions known a priori, as a method to detect curbstoning. By exploiting common deficiencies in statistical understanding, survey collectors imputing data to these questions can leverage empirically known distributions to determine deviation from the expected distribution of responses. We examined both authentic and fabricated surveys that included these questions and we compared the observed distributions with the expected distributions. The majority of the proposed methods had Type I error rates near or below the specified alpha level (.05). The methods demonstrated the ability to detect false responses correctly 48%-90% of the time across two samples when surveying at least 50 participants. While the methods varied in effectiveness, combining these methods demonstrated the highest statistical power, with Type I error rates lower than 1%. Additionally, even in situations with smaller sample sizes (e.g., N = 30), combining these methods allows them to be effective in detecting curbstoning. These methods provide a simple and generalizable way for researchers not present during data collection to possess accurate data.

show abstract

Section: Educating Interviewersmentioning

confidence: 99%

Curbing curbstoning: Distributional methods to detect survey data fabrication by third-parties.

Hernandez¹,

Ristow²,

Hauenstein³

2022

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…Early strategies include asking enumerators to sign statements airming that they correctly followed protocols (Bennett 1948). Perhaps the most efective, and most expensive, strategy for assuring data quality is the "callback," where ieldwork supervisors conduct partial re-interviews with participants to verify their participation (Biemer and Stokes 1989;Schäfer et al 2004;Stokes and Jones 1989;Swanson, Cho, and Eltinge 2003;Winker 2016). More recently, researchers have introduced checks for interview duplication and straightlining, wherein enumerators or staf at survey irms generate fraudulent interviews by illing out identical answers across a battery of questions (Blasius 2018;Blasius and Thiessen 2012Simmons et al 2016;Slomczynski, Powalko, and Krauze 2017).…”

Section: Strategies For Preventing Low-quality Interviewsmentioning

confidence: 99%

How to Get Better Survey Data More Efficiently

Cohen¹,

Warner²

2020

Polit. Anal.

View full text Add to dashboard Cite

A key challenge facing many large, in-person public opinion surveys is ensuring that enumerators follow fieldwork protocols. Implementing “quality control” processes can improve data quality and help ensure the representativeness of the final sample. Yet while public opinion researchers have demonstrated the utility of quality control procedures such as audio capture and geo-tracking, there is little research assessing the relative merits of such tools. In this paper, we present new evidence on this question using data from the 2016/17 wave of the AmericasBarometer study. Results from a large classification task demonstrate that a small set of automated and human-coded variables, available across popular survey platforms, can recover the final sample of interviews that results when a full suite of quality control procedures is implemented. Taken as a whole, our results indicate that implementing and automating just a few of the many quality control procedures available can streamline survey researchers’ quality control processes while substantially improving the quality of their data.

show abstract

“…Interviewers may decide to falsify because they want to optimize their cost-benefit balance. Thus, they reduce time and effort, particularly when their objective is difficult, for example reaching a certain difficult-to-reach target person and obtaining participation (Sodeur, 2007; Turner et al, 2002; Winker, 2016).…”

Section: State Of Research On Interviewer Falsificationsmentioning

confidence: 99%

Investigating Interviewer Falsifications – A Quasi-experimental Design

Landrock

2017

Bulletin of Sociological Methodology/Bulletin de Méthodologie S

View full text Add to dashboard Cite

This paper discusses the strengths and limitations of experimentally-orientated research in sociology. In principal, results from experiments cannot be generalized. Nevertheless they allow controlling for certain conditions that are impossible to measure during a survey. This paper gives an example: It presents findings from a quasi-experimental research on interviewer falsifications. Interviewers conducted real standardized interviews and subsequently falsified corresponding interviews in the lab. This enables a comparison of real and falsified survey data. Our research design has the limitation that there is no proof that actual falsifiers in an actual survey fieldwork environment would have falsified interviews in the same way as did the participants of our study. However, only this quasi-experimental approach allows us to know for sure which interviews are falsified.

show abstract

Assuring the quality of survey data: Incentives, detection and documentation of deviant behavior

Cited by 8 publications

References 20 publications

Curbing curbstoning: Distributional methods to detect survey data fabrication by third-parties.

Curbing curbstoning: Distributional methods to detect survey data fabrication by third-parties.

How to Get Better Survey Data More Efficiently

Investigating Interviewer Falsifications – A Quasi-experimental Design

Contact Info

Product

Resources

About