Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program

Pengra, Bruce W.; Stehman, Stephen V.; Horton, Josephine A.; Dockter, Daryn; Schroeder, Todd A.; Yang, Zhiqiang; Cohen, Warren B.; Healey, Sean P.; Loveland, Thomas R.

doi:10.1016/j.rse.2019.111261

Cited by 61 publications

(101 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This uncertainty should be measured and factored into the accuracy assessment [39,46]. However, in practice this accounting is rarely done, while map reference data uncertainty is also rarely examined [34,38,57]. This tendency is illustrated by Ye et al [65], who reviewed 209 journal articles focused on object-based image analysis, finding that one third gave incomplete information about the sample design and size of their map reference data, let alone any mention of error within the sample.…”

Section: Map Accuracy Assessment Proceduresmentioning

confidence: 99%

“…The first of these are errors of interpretation [39], which are mistakes created in the process of manual image interpretation. Image interpretation is widely used to generate TD, but often this technique leads to inconsistent labels between interpreters for the same areas of interest [34,37,99,155]. Interpreters may lack experience in the task or be unfamiliar with the context of the study area, e.g., [156].…”

Section: Collection-related Errorsmentioning

confidence: 99%

“…Errors in these products also impact the veracity of any downstream products into which they are ingested [33]. While progress in algorithmic performance continues apace, standards regarding the collection and use of TD remain uncoordinated across researchers [34]. Additionally, much of the research and development of big data and ML is occurring in industry and the fields of computer science and (non-spatial) data science, leaving a potential knowledge gap for EO scientists [35,36].…”

Section: Introductionmentioning

confidence: 99%

“…Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.3 of 39 user groups [37][38][39][40][41][42][43][44][45]. This includes substantial work to measure error in map reference data (i.e., the independent sample used to assess map accuracy) and account for its impact on map assessment [34,38,46,47]. However, focus on the quality and impacts of TD error has been less systematic.…”

mentioning

confidence: 99%

“…Previous research shows that the impact of TD error can be substantial but varied, suggesting that a more comprehensive approach to this issue is warranted. Furthermore, while TD and map reference data are often collected using the same approaches [55][56][57] and often subject to the same errors, the existing procedures to minimize and account for map reference errors [34,38,46,47] are not necessarily relevant for quantifying the impacts of TD error. The problems associated with TD error can be summarized as follows:1.The "big data" era vastly increases the demand for TD.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

et al. 2020

View full text Add to dashboard Cite

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter is particularly relevant for image-interpreted TD, an increasingly commonly used method due to its practicality and the increasing training sample size requirements of modern ML algorithms. TD errors can cause substantial errors in the maps created using ML algorithms, which may impact map use and interpretation. Despite these potential errors and their real-world consequences for map-based decisions, TD error is often not accounted for or reported in EO research. Here we review the current practices for collecting and handling TD. We identify the sources of TD error, and illustrate their impacts using several case studies representing different EO applications (infrastructure mapping, global surface flux estimates, and agricultural monitoring), and provide guidelines for minimizing and accounting for TD errors. To harmonize terminology, we distinguish TD from three other classes of data that should be used to create and assess ML models: training reference data, used to assess the quality of TD during data generation; validation data, used to iteratively improve models; and map reference data, used only for final accuracy assessment. We focus primarily on TD, but our advice is generally applicable to all four classes, and we ground our review in established best practices for map accuracy assessment literature. EO researchers should start by determining the tolerable levels of map error and appropriate error metrics. Next, TD error should be minimized during sample design by choosing a representative spatio-temporal collection strategy, by using spatially and temporally relevant imagery and ancillary data sources during TD creation, and by selecting a set of legend definitions supported by the data. Furthermore, TD error can be minimized during the collection of individual samples by using consensus-based collection strategies, by directly comparing interpreted training observations against expert-generated training reference data to derive TD error metrics, and by providing image interpreters with thorough application-specific training. We strongly advise that TD error is incorporated in model outputs, either directly in bias and variance estimates or, at a minimum, by documenting the sources and implications of error. TD should be fully documented and made available via an open TD repository, allowing others to replicate and assess its use. To guide researchers in this process, we propose three tiers of TD error accounting standards. Finally, we advise researchers to clearly communicate the magnitude and impacts of TD error on map outputs, with specific consideration given to the likely map audience.

show abstract

Section: Map Accuracy Assessment Proceduresmentioning

confidence: 99%

Section: Collection-related Errorsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

et al. 2020

View full text Add to dashboard Cite

show abstract

Diverse migratory portfolios drive inter‐annual switching behavior of elk across the Greater Yellowstone Ecosystem

et al. 2023

View full text Add to dashboard Cite

A growing body of evidence shows that some ungulates alternate between migratory and nonmigratory behaviors over time. Yet it remains unclear whether such short‐term behavioral changes can help explain reported declines in ungulate migration worldwide, as opposed to long‐term demographic changes. Furthermore, advances in tracking technology reveal that a simple distinction between migration and nonmigration may not sufficiently describe all individual behaviors. To better understand the dynamics and drivers of ungulate switching behavior, we investigated 14 years of movement data from 361 elk in 20 herds across the Greater Yellowstone Ecosystem (GYE). First, we categorized yearly individual behaviors using a clustering algorithm that identified similar migratory tactics across a continuum of behaviors. Then, we tested seven hypotheses to explain why some ungulates switch behaviors, and we evaluated how behavioral changes affected the proportions of different behaviors across the system. We identified four distinct behavioral tactics: residents (4.8% of elk‐years), short‐distance migrants (53.7%), elevational migrants (21.9%) and long‐distance migrants (19.6%). Of the 20 herds, 18 were partially migratory, and 5 had all four movement tactics present. We observed switches between migratory tactics in all sets of consecutive years during our study period, with an average of 22.5% of individual elk changing movement tactics from one year to the next. Elk in herds with higher movement tactic diversity were significantly more likely to switch tactics and often responded more effectively to adverse environmental conditions, compared to those in herds with low movement tactic diversity. During our study period, switching increased the prevalence of both short‐ and long‐distance migrants, decreased the prevalence of elevational migrants, and had no effect on the prevalence of residents. Our findings suggest that rather than contributing to the declining migratory behavior found in the GYE, switching behavior may enable greater resiliency to continuously changing environmental and anthropogenic conditions.

show abstract

Remote Sensing of Land Change: A Multifaceted Perspective

Zhu¹,

Qiu²,

Su³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program

Cited by 61 publications

References 30 publications

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Diverse migratory portfolios drive inter‐annual switching behavior of elk across the Greater Yellowstone Ecosystem

Remote Sensing of Land Change: A Multifaceted Perspective

Contact Info

Product

Resources

About