Making inference with messy (citizen science) data: when are data accurate enough and how can they be improved?

Abstract

Measurement or observation error is common in ecological data: as citizen scientists and automated algorithms play larger roles processing growing volumes of data to address problems at large scales, concerns about data quality and strategies for improving it have received greater focus. However, practical guidance pertaining to fundamental data quality questions for data users or managers—how accurate do data need to be and what is the best or most efficient way to improve it?—remains limited. We present a generalizable framework for evaluating data quality and identifying remediation practices, and demonstrate the framework using trail camera images classified using crowdsourcing to determine acceptable rates of misclassification and identify optimal remediation strategies for analysis using occupancy models. We used expert validation to estimate baseline classification accuracy and simulation to determine the sensitivity of two occupancy estimators (standard and false‐positive extensions) to different empirical misclassification rates. We used regression techniques to identify important predictors of misclassification and prioritize remediation strategies. More than 93% of images were accurately classified, but simulation results suggested that most species were not identified accurately enough to permit distribution estimation at our predefined threshold for accuracy (<5% absolute bias). A model developed to screen incorrect classifications predicted misclassified images with >97% accuracy: enough to meet our accuracy threshold. Occupancy models that accounted for false‐positive error provided even more accurate inference even at high rates of misclassification (30%). As simulation suggested occupancy models were less sensitive to additional false‐negative error, screening models or fitting occupancy models accounting for false‐positive error emerged as efficient data remediation solutions. Combining simulation‐based sensitivity analysis with empirical estimation of baseline error and its variability allows users and managers of potentially error‐prone data to identify and fix problematic data more efficiently. It may be particularly helpful for “big data” efforts dependent upon citizen scientists or automated classification algorithms with many downstream users, but given the ubiquity of observation or measurement error, even conventional studies may benefit from focusing more attention upon data quality.

Integrating remote sensing and jurisdictional observation networks to improve the resolution of ecological management

“…Overall, local colonization appears to be slightly more likely than long distance colonization, and local colonization may lead to more stable growth because persistence appears to be more likely with occupied neighboring cells. Based upon previous work (Clare et al 2019, C. Anhalt-Depies unpublished data), the classification accuracy of black bears within images is believed to be very high (> 99% accuracy). Although the data used for fitting may include some false positives, we ignore such error here given previous evidence that its incidence is not likely to substantially skew model predictions and estimates.…”

“…For example, recent work (C. Anhalt-Depies, unpublished data) has suggested that camera host classifications of striped skunks are sufficiently accurate (> 97%) that there is little benefit associated with subjecting these images to crowdsourcing, while gray foxes are inaccurately classified by both camera hosts and via crowdsourcing, and may generally require expert review. Similarly, while Clare et al (2019) found that coyotes were relatively commonly misclassified via crowdsourcing, follow up work suggests that both camera hosts and crowdsourcing more accurately identify coyotes than previously recognized (C. Anhalt-Depies, unpublished data).…”

“…Only images subject to crowdsourced classification from Zooniverse are tagged with behaviors. All images used for analysis were filtered such that we only used images in which one individual deer was in the image (to avoid modeling complications associated with multiple animals) and such that the predicted accuracy of the species classification was > 90% (following Clare et al 2019; purpose being to avoid behavioral classifications based upon a limited number of votes). All images were drawn from 2017.…”

Correcting Misclassification Errors in Crowdsourced Ecological Data: A Bayesian Perspective

“…These platforms connect millions of collaborators all over the world. However, one of the main concerns when making statistical inferences using data obtained via crowdsourcing is the inherent presence of misclassification or measurement errors resulting from participants’ variable skill levels and abilities (Bachrach et al., 2012; Bird et al., 2014; Clare et al., 2019; Mengersen et al., 2017; Venanzi et al., 2014). A second concern relates to spatial dependence in the data, which has been found to produce incorrect estimates in species abundance models when it is not accounted for (e.g.…”

“…Within the occupancy modelling framework, several authors have approached the issue of bias correction by means of performance measures, especially the false‐positive rates (Chambert et al., 2015; Clare et al., 2019). A recent extension suggested by Pacifici et al.…”