Conflation of expert and crowd reference data to validate global binary thematic maps

Waldner, François; Schucknecht, Anne; Lesiv, Myroslava; Gallego, Javier; See, Linda; Pérez-Hoyos, Ana; d’Andrimont, Raphaël; Maet, Thomas De; Bayas, Juan Carlos Laso; Fritz, Steffen; Léo, Oberdan; Herve, Kerdiles; Díez, Mónica; Tricht, Kristof Van; Gilliams, Sven; Shelestov, Andrii; Lavreniuk, Mykola; Simões, Margareth; Ferraz, R. P. D.; Bellón, Beatriz; Bégué, Agnès; Hazeu, G.W.; Stonacek, Vaclav; Kolomazník, Jan; Misurec, J; Verón, Santiago R.; Abelleyra, Diego de; Плотников, Д.Е.; Li, Mingyong; Singha, Mrinal; Patil, P. G.; Zhang, Miao; Defourny, Pierre

doi:10.1016/j.rse.2018.10.039

Cited by 25 publications

(23 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Medium-resolution time series or high-resolution snapshot images are also at hand to facilitate the labeling process. The interface was recently simplified to allow experts in remote sensing and agronomy from the EU-funded SIGMA project 4 to collect binary cropland data at 4147 locations (Waldner et al 2018). Unfortunately, the reference data that have been collected through this bespoke interface over several years have not been made openly available.…”

Section: Other Toolsmentioning

confidence: 99%

Recent Advances in Forest Observation with Visual Interpretation of Very High-Resolution Imagery

et al. 2019

Self Cite

View full text Add to dashboard Cite

The land area covered by freely available very high-resolution (VHR) imagery has grown dramatically over recent years, which has considerable relevance for forest observation and monitoring. For example, it is possible to recognize and extract a number of features related to forest type, forest management, degradation and disturbance using VHR imagery. Moreover, time series of medium-to-high-resolution imagery such as MODIS, Landsat or Sentinel has allowed for monitoring of parameters related to forest cover change. Although automatic classification is used regularly to monitor forests using medium-resolution imagery, VHR imagery and changes in web-based technology have opened up new possibilities for the role of visual interpretation in forest observation. Visual interpretation of VHR is typically employed to provide training and/or validation data for other remote sensing-based techniques or to derive statistics directly on forest cover/forest cover change over large regions. Hence, this paper reviews the state of the art in tools designed for visual interpretation of VHR, including Geo-Wiki, LACO-Wiki and Collect Earth as well as issues related to interpretation of VHR imagery and approaches to quality assurance. We have also listed a number of success stories where visual interpretation plays a crucial role, including a global forest mask harmonized with FAO FRA country statistics; estimation of dryland forest area; quantification of deforestation; national reporting to the UNFCCC; and drivers of forest change.

show abstract

Section: Other Toolsmentioning

confidence: 99%

Recent Advances in Forest Observation with Visual Interpretation of Very High-Resolution Imagery

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another design-related problem arises from large-scale data collection initiatives that are becoming increasingly common due to the expanding extent of modern EO analyses, e.g., [148]. These efforts, often conducted via crowdsourcing campaigns, typically enlist citizens to collect data via a web-based platform, e.g., [66,[149][150][151]. Examples include OSM, Geo-Wiki [66], Collect Earth [152], DIYLandcover [150], and FotoQuest Go [153].…”

Section: Design-related Errorsmentioning

confidence: 99%

“…Once crowdsourced/citizen science data have been post-processed for noise, they can be highly detailed and spatially extensive [66,[69][70][71]. Nevertheless, quality problems in such datasets can be particularly hard to find and clean and are thus an important source of TD error that may propagate through ML algorithms into map outputs [57,151,176]. Therefore, these data should be used more cautiously than expert-derived TD.…”

Section: Collection-related Errorsmentioning

confidence: 99%

“…Aside from these problems, there are many sources of technologically induced errors, such as defects in the software or hardware of measurement devices, user input error, or calibration errors (e.g., in spectroradiometers or other equipment). However, accounting for quantitative measurement error is more Collection-related errors may be particularly acute in large-scale crowdsourcing campaigns or citizen science initiatives, which are increasingly valued for mapping projects due to their larger size and cheaper acquisition costs [22,66,150,151]. Such datasets are often collected rapidly and entail labeling many observations over a short period of time by participants who are not domain experts [153,174].…”

Section: Collection-related Errorsmentioning

confidence: 99%

See 1 more Smart Citation

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

et al. 2020

View full text Add to dashboard Cite

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

show abstract

“…These rules are applied in a two-step process: the first step is performed by a photo-interpreter who assigns a label to each sub-part, and the second step consists in aggregating the labels of the sub-parts to attribute a final label, which can be automated. While square sub-parts are the most widespread type of partition (Bayas et al, 2017), irregular polygons can also be used (Waldner et al, 2019). In the latter case, accurate delineation of the polygons plays a major role in the reliability of the response design.…”

Section: Partition-based Response Designsmentioning

confidence: 99%

How Response Designs and Class Proportions Affect the Accuracy of Validation Data

2020

Self Cite

View full text Add to dashboard Cite

Reference data collected to validate land cover maps are generally considered free of errors. In practice, however, they contain errors despite all efforts to minimise them. These errors then propagate up to the accuracy assessment stage and impact the validation results. For photo-interpreted reference data, the three most widely studied sources of error are systematic incorrect labelling, vigilance drops, and demographic factors. How internal estimation errors, i.e., errors intrinsic to the response design, affect the accuracy of reference data is far less understood. In this paper, we analysed the impact of estimation errors for two types of legends (binary and multi-class) as well as for two common response designs (point-based and partition-based) with a range of sub-sample sizes. We showed that the accuracy of response designs depends on the class proportions within the sampling units, with complex landscapes being more prone to errors. As a result, response designs where the number of sub-samples are fixed are inefficient, and the labels of reference data sets have inconsistent confidence levels. In order to control estimation errors, to guarantee high accuracy standards of validation data, and to minimise data collection efforts, we proposed to rely on confidence intervals of the photo-interpreted data to define how many sub-samples should be labelled. In practice, sub-samples are iteratively selected and labelled until the estimated class proportions reach the desired level of confidence. As a result, less effort is spent on labelling obvious cases and the spared effort can be allocated to more complex cases, thus leading to an increased reliability of reference data and in turn better accuracy assessment. Across our study site, we demonstrated that such an approach could reduce the labelling effort by 50% to 75%, with greater gains in homogeneous landscapes. We contend that adopting this optimisation approach will not only increase the efficiency of reference data collection but will also help deliver reliable accuracy estimates to the user community.

show abstract

Conflation of expert and crowd reference data to validate global binary thematic maps

Cited by 25 publications

References 50 publications

Recent Advances in Forest Observation with Visual Interpretation of Very High-Resolution Imagery

Recent Advances in Forest Observation with Visual Interpretation of Very High-Resolution Imagery

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

How Response Designs and Class Proportions Affect the Accuracy of Validation Data

Contact Info

Product

Resources

About