Managing Data Quality Indicators with Paradata Based Statistical Quality Control Tools: The Keys to Survey Performance

Jans, Matt; Sirkis, Robyn; Morgan, David

doi:10.1002/9781118596869.ch9

Cited by 11 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore, only interviews in an outlying week are further checked. This is in line with the goal of statistically controlling a process, instead of performing 100% inspection (Jans et al, 2013).…”

Section: Drilling Down From An Outlying Week To Individual Interviewssupporting

confidence: 68%

“…First, the grouping of interviews by weeks limits the use of S charts. In survey settings, a relevant time scale (such as fieldwork days and weeks) is recommended to group the interviews (Jans et al, 2013). A related problem is that interviews can be distributed rather unevenly over time.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

“…Despite the relevance that the monitoring and evaluating of paradata can have to the assessment of data quality, only a few researchers have applied SPC techniques in survey settings, and have focused on the post-survey use of these techniques. Sirkis, Jans, Dahlhamer, Gindi, and Duffey (2011) and Jans, Sirkis, and Morgan (2013) demonstrated the use of control charts to analyze interview pace for the U.S. National Health Interview Survey (January 2008 to December 2010). They found SPC techniques implementable and useful, because interviewers who needed further examination were identified for survey supervisors.…”

Section: Introductionmentioning

confidence: 99%

“…An exceptionally long interview, on the other hand, could indicate a respondent being uncertain about the answers and therefore less likely to give "correct" responses (Jans et al, 2013;Olson & Peytchev, 2007). Researchers have reported that respondents with longer response times had a lower probability of giving a correct answer 1 in the context of web surveys (Heerwegh, 2003) and computer-assisted telephone interviewing (Draisma & Dijkstra, 2004).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

2019

View full text Add to dashboard Cite

Despite general agreement regarding the usefulness of statistical process control (SPC) tools for monitoring paradata, using SPC from an early phase of the survey fieldwork is rather rare. This study focuses on one type of paradata-interview duration-to fill this void. First, we establish a procedure based on the idea of enabling fieldwork monitoring for the seventh round of the European Social Survey in Belgium from its start. The impact of respondent characteristics on interview duration is controlled for by multiple regression. Moreover, we simulate the real conditions of an ongoing survey data collection process by cumulating data and repeating the identification of problematic interviews each week, on the basis that "new" data are available. Second, for each interview we record and track the results with regard to whether or not it is problematic over the fieldwork period, to examine the consistency of our findings. We find that as more data becomes available, the results concerning whether an interview is problematic changes in only 0.3% of the cases. Out of the 27 interviews identified as problematic when all information was available, 25 were immediately identified once relevant information was available. Overall, these findings suggest that SPC tools are reliable and efficient in a survey context, and accordingly have great potential for allowing survey practitioners to focus on the interviews for which further examination is needed immediately, rather than when the data collection has been completed.

show abstract

Section: Drilling Down From An Outlying Week To Individual Interviewssupporting

confidence: 68%

Section: Conclusion and Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

2019

View full text Add to dashboard Cite

show abstract

“…In the field of surveys, previous work has explored the use of various types of control charts, such as the Xbar/R chart (Peng and Feld 2011;Reed and Reed 1997), Xbar/S chart (Sirkis and Jans 2011;Jans et al 2013;Jin et al 2019), p chart (Pierchala andSurti 1999;Corsetti et al 2010), and EWMA control chart (Peng and Feld 2011). As noted earlier, all of these charts focus on one single variable.…”

Section: Introductionmentioning

confidence: 99%

Assessing Response Quality by Using Multivariate Control Charts for Numerical and Categorical Response Quality Indicators

Jin

Loosveldt

2020

Journal of Survey Statistics and Methodology

View full text Add to dashboard Cite

When assessing interview response quality to identify potentially low-quality interviews, both numerical and categorical response quality indicators (mixed indicators) are usually available. However, research on how to use them simultaneously is very rare. In the current article, we extend the application of conventional multivariate control charts to include response quality indicators that are of a mixed type. We analyze data from the eighth round of the European Social Survey in Belgium, characterized by six numerical and two categorical response quality indicators. First, we employ a principal component analysis mix procedure (PCA Mix) to transform the mixed quality indicators into principal components. The principal component scores are subsequently used to construct a Hotelling T2 statistic. To deal with the non-multivariate normal nature of the principal component scores obtained from the PCA Mix, a nonparametric bootstrap method is then applied to calculate the control limit for the T2 statistic. Second, we suggest tools to interpret an identified outlier in terms of finding the responsible original indicator(s). Third, we present a cyclic procedure for determining the “in-control” data, by iteratively removing the outliers until the process is considered as being in control. Lastly, we identify the most important indicators that discriminate the outliers from the in-control data. Our results imply that multivariate control charts based on relevant projection tools such as PCA Mix in combination with the bootstrap technique have great potential for use in evaluating interview response quality and identifying outliers.

show abstract

Calibration and Other Uses of Auxiliary Data in Weighting

Valliant

Dever

Kreuter

2013

Practical Tools for Designing and Weighting Survey Samples

View full text Add to dashboard Cite

Managing Data Quality Indicators with Paradata Based Statistical Quality Control Tools: The Keys to Survey Performance

Cited by 11 publications

References 18 publications

The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

The Evaluation of Statistical Process Control Methods to Monitor Interview Duration During Survey Data Collection

Assessing Response Quality by Using Multivariate Control Charts for Numerical and Categorical Response Quality Indicators

Calibration and Other Uses of Auxiliary Data in Weighting

Contact Info

Product

Resources

About