Automated quality control for mobile data collection

Abstract

Systematic interviewer error is a potential issue in any health survey, and it can be especially pernicious in low-and middleincome countries, where survey teams may face problems of limited supervision, chaotic environments, language barriers, and low literacy. Survey teams in such environments could benefit from software that leverages mobile data collection tools to provide solutions for automated data quality control. As a first step in the creation of such software, we investigate and test several algorithms that find anomalous patterns in data. We validate the algorithms using one labeled data set and two unlabeled data sets from two community outreach programs in East Africa. In the labeled set, some of the data is known to be fabricated and some is believed to be relatively accurate. The unlabeled sets are from actual field operations. We demonstrate the feasibility of tools for automated data quality control by showing that the algorithms detect the fake data in the labeled set with a high sensitivity and specificity, and that they detect compelling anomalies in the unlabeled sets.

Using behavioral data to identify interviewer fabrication in surveys

“…More recently, researchers have begun to investigate the ability of supervised classification to predict fabrication or other data quality problems [4,10,31]. These papers show the promise of the approach, but, unlike our work, they do not investigate the potential of low-level behavioral data to improve performance.…”

“…4 On data set D 0 , the accuracy was 96%; on data set D 1 , the accuracy was 90%; and on data set D 2 , the accuracy was 86%. 5 More detailed performance measures-including the precision, recall, specificity, and F 1 -score-are shown in Table 6.…”

“…We chose this value for all of our experiments. To optimize performance, we initially performed a search through values of K(2,4,8,16,32, 64, 128), using as an optimization criterion the accuracy achieved during 10-fold cross-validation on the training set (averaged over 10 repetitions). Once we did this, we saw that performance did not depend strongly on K. Thus, for simplicity, we set K = 32 in the remainder of our experiments.…”

DevCAFE 1.0: A participatory platform for assessing development initiatives in the field

“…While extremely useful, such approaches are still limited in the types of data they can collect. A related point of ensuring the quality of collected data has also been studied [5,9,29].…”

“…Digital survey tools are useful for experimental designs and confirmatory analysis as they mitigate sources of bias [5]. However, development participants, practitioners and researchers are often interested in participatory, exploratory analysis to discover patterns, problems and issues of interest while in the field [16].…”