Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts

Abstract

There is currently a lack of in-situ environmental data for the calibration and validation of remotely sensed products and for the development and verification of models. Crowdsourcing is increasingly being seen as one potentially powerful way of increasing the supply of in-situ data but there are a number of concerns over the subsequent use of the data, in particular over data quality. This paper examined crowdsourced data from the Geo-Wiki crowdsourcing tool for land cover validation to determine whether there were significant differences in quality between the answers provided by experts and non-experts in the domain of remote sensing and therefore the extent to which crowdsourced data describing human impact and land cover can be used in further scientific research. The results showed that there was little difference between experts and non-experts in identifying human impact although results varied by land cover while experts were better than non-experts in identifying the land cover type. This suggests the need to create training materials with more examples in those areas where difficulties in identification were encountered, and to offer some method for contributors to reflect on the information they contribute, perhaps by feeding back the evaluations of their contributed data or by making additional training materials available. Accuracies were also found to be higher when the volunteers were more consistent in their responses at a given location and when they indicated higher confidence, which suggests that these additional pieces of information could be used in the development of robust measures of quality in the future.

Crowdsourcing, Citizen Science or Volunteered Geographic Information? The Current State of Crowdsourced Geographic Information

“…The quality of citizen derived data can be viewed from a variety of ways [57]. Many comparative studies have shown that crowdsourced geographic information can be as good, if not better, than data from authoritative sources [58,59]. A comprehensive literature overview of the latest developments in crowdsourced geographic information research is presented in Reference [60], with a focus on trends related to OpenStreetMap while many others have discussed the quality of this volunteer data source [61][62][63].…”

“…The majority of sites reviewed fell into the first two categories, which implies that very little analysis of the crowdsourced geographic information can be undertaken in relation to the background of individuals. Some exceptions include research on contributors to OpenStreetMap [73] and Geo-Wiki [58].…”

The Impact of Contributor Confidence, Expertise and Distance on the Crowdsourced Land Cover Data Quality

“…This paper extends the research described in See et al (2013) by considering the impact of distance between the volunteer's location and the location being analysed. It shows that distance has a minor effect on the reliability of the labelling performed by volunteers, and that expertise matters generally but not for all classes such as Shrub Cover.…”

Crowdsourcing Design: A Synthesis of Literatures

“…When simple and clear instruction are provided to the crowds, See et al (2013) found that the crowds can improve the quality of information faster than the experts. This indicates the importance of designing effective means to aggregate information from the crowds.…”