Quality assessment of classification and cluster maps without ground truth knowledge

Baraldi, A.; Bruzzone, Lorenzo; Blonda, P.

doi:10.1109/tgrs.2004.843074

Cited by 74 publications

(53 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Liu and Zhou, 2004;Lu et al, 2008) and some even seek to work without ground data (Baraldi et al, 2005;Bruzzone and Marconcini, 2009). Moreover, it is sometimes noted that problems with the ground reference data may have negative impacts on the apparent accuracy of maps derived by remote sensing .…”

Section: Ground Data and Their Accuracymentioning

confidence: 99%

Assessing the accuracy of land cover change with imperfect ground reference data

Foody

2010

Remote Sensing of Environment

273

159

View full text Add to dashboard Cite

The manuscript of the above article revised after peer review and submitted to the journal for publication, follows. Please note that small changes may have been made after submission and the definitive version is that subsequently published as: AbstractThe ground data used as a reference in the validation of land cover change products are often not an ideal gold standard but degraded by error. The effects of ground reference data error on the accuracy of land cover change detection and the accuracy of estimates of the extent of change was evaluated. Twelve data sets were simulated to allow the exploration of the impacts of a spectrum of ground data imperfections on the estimation of the producer's and user's accuracy of change as well as of change extent. Simulated data were used since this ensured that the actual properties of the data were known and to exclude effects due to other sources of ground reference data error; although the impacts of simulated reference data on two real confusion matrices is also illustrated. The imperfections evaluated ranged from the inclusion of small amounts of known error into the ground reference data through to the extreme situation in which ground data were absent. The results show that even small amounts of error in the ground reference data can introduce large error into studies of land cover change by remote sensing and reinforce the desire to avoid the expression ground truth as this might imply that the data are a gold standard reference. The effect of reference data imperfections was dependent on the degree of association between the errors in the cross tabulated data sets. For 3 example, in the scenarios investigated, a 10% error in the reference data set introduced an under-estimation of the producer's accuracy of 18.5% if the errors were independent but an over-estimation of the producer's accuracy of 12.3% if the errors were correlated. The magnitude of the mis-estimation of the producer's accuracy was also a function of the amount of change and greatest at low levels of change. The amount of land cover change estimated also varied greatly as a function of ground reference data error. Some possible methods to reduce or even remove the impacts of ground reference data error were illustrated. These ranged from simple algebraic means to estimate the actual values of accuracy and change extent if the imperfections were known through to a latent class analysis that allowed the assessment of classification accuracy and estimation of change extent without the use of ground reference data if the underlying model is defined appropriately.

show abstract

Section: Ground Data and Their Accuracymentioning

confidence: 99%

Assessing the accuracy of land cover change with imperfect ground reference data

Foody

2010

Remote Sensing of Environment

273

159

View full text Add to dashboard Cite

show abstract

“…As a consequence, the output product of a traditional data clustering or image segmentation algorithm falls in the class of preliminary maps or primal sketches of an input MS image [22]- [26]. Vice versa, the output product of a supervised classifier (e.g., a plug-in maximum-likelihood classifier and a supervised inductive learning MLP), which is known as classification (i.e., thematic and informational) map consisting of land cover classes, is not a preliminary map of an input MS image.…”

Section: Abstract and Definitionsmentioning

confidence: 99%

Automatic Spectral Rule-Based Preliminary Mapping of Calibrated Landsat TM and ETM+ Images

Baraldi

Puzzolo²,

Blonda

et al. 2006

IEEE Trans. Geosci. Remote Sensing

Self Cite

104

149

View full text Add to dashboard Cite

Abstract-Based on purely spectral-domain prior knowledge taken from the remote sensing (RS) literature, an original spectral (fuzzy) rule-based per-pixel classifier is proposed. Requiring no training and supervision to run, the proposed spectral rule-based system is suitable for the preliminary classification (primal sketch, in the Marr sense) of Landsat-5 Thematic Mapper and Landsat-7 Enhanced Thematic Mapper Plus images calibrated into planetary reflectance (albedo) and at-satellite temperature. The classification system consists of a modular hierarchical top-down processing structure, which is adaptive to image statistics, computationally efficient, and easy to modify, augment, or scale to other sensors' spectral properties, like those of the Advanced Spaceborne Thermal Emission and Reflection Radiometer and of the Satellite Pour l'Observation de la Terre (SPOT-4 and -5). As output, the proposed system detects a set of meaningful and reliable fuzzy spectral layers (strata) consistent (in terms of one-to-one or manyto-one relationships) with land cover classes found in levels I and II of the U.S. Geological Survey classification scheme. Although kernel spectral categories (e.g., strong vegetation) are detected without requiring any reference sample, their symbolic meaning is intermediate between those (low) of clusters and segments and those (high) of land cover classes (e.g., forest). This means that the application domain of the kernel spectral strata is by no means alternative to RS data clustering, image segmentation, and land cover classification. Rather, prior knowledge-based kernel spectral categories are naturally suitable for driving stratified application-specific classification, clustering, or segmentation of RS imagery that could involve training and supervision. The efficacy and robustness of the proposed rule-based system are tested in two operational RS image classification problems.Index Terms-Data clustering, fuzzy rule, fuzzy set (FS), generalization capability, image classification, image color analysis, image segmentation, one-class classifier, prior knowledge, remotely sensed imagery, spectral information, supervised and unsupervised learning from finite data.

show abstract

“…In general the class of (square and sorted) CMTRX instances is a special case of the class of OAMTRX instances, either square or non-square, i.e., OAMTRX ⊃ CMTRX. A similar consideration holds about summary Q 2 Is generated from an OAMTRX or a CMTRX, i.e., Q 2 I(OAMTRX) ⊃ Q 2 I(CMTRX) (Baraldi et al, 2014, 2005, 2006). …”

Section: Problem Background Of Color Naming In Cognitive Sciencementioning

confidence: 99%

“…In greater detail, for any BIVRFTAB instance, either square or non-square, there is a binary relationship R: A ⇒ B ⊆ A × B that guides the interpretation process, where “correct” binary entry-pair cells of the 2-fold Cartesian product A × B are equal to 1 and located either off-diagonal (scattered) or on-diagonal, if a main diagonal exists when the BIVRFTAB is square. When a BIVRFTAB is estimated from a geospatial population with or without sampling, it is called overlapping area matrix (OAMTRX) (Baraldi et al, 2014; Baraldi, Bruzzone, & Blonda, 2005; Baraldi et al, 2006; Beauchemin & Thomson, 1997; Lunetta & Elvidge, 1999; Ortiz & Oliver, 2006; Pontius & Connors, 2006). When the binary relationship R: A ⇒ B is a bijective function (both 1–1 and onto), i.e., when the two categorical variables A and B estimated from a single population coincide, then the BIVRFTAB instantiation is square and sorted; it is typically called confusion matrix (CMTRX) or error matrix (Congalton & Green, 1999; Lunetta & Elvidge, 1999; Pontius & Millones, 2011; Stehman & Czaplewski, 1998).…”

Section: Problem Background Of Color Naming In Cognitive Sciencementioning

confidence: 99%

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation - Part 1: Theory

et al. 2018

View full text Add to dashboard Cite

ESA defines as Earth Observation (EO) Level 2 information product a single-date multi-spectral (MS) image corrected for atmospheric, adjacency and topographic effects, stacked with its data-derived scene classification map (SCM), whose legend includes quality layers cloud and cloud-shadow. No ESA EO Level 2 product has ever been systematically generated at the ground segment. To fill the information gap from EO big data to ESA EO Level 2 product in compliance with the GEO-CEOS stage 4 validation (Val) guidelines, an off-the-shelf Satellite Image Automatic Mapper (SIAM) lightweight computer program was validated by independent means on an annual 30 m resolution Web-Enabled Landsat Data (WELD) image composite time-series of the conterminous U.S. (CONUS) for the years 2006–2009. The SIAM core is a prior knowledge-based decision tree for MS reflectance space hyperpolyhedralization into static color names. Typically, a vocabulary of MS color names in a MS data (hyper)cube and a dictionary of land cover (LC) class names in the scene-domain do not coincide and must be harmonized (reconciled). The present Part 1—Theory provides the multidisciplinary background of a priori color naming. The subsequent Part 2—Validation accomplishes a GEO-CEOS stage 4 Val of the test SIAM-WELD annual map time-series in comparison with a reference 30 m resolution 16-class USGS National Land Cover Data 2006 map, based on an original protocol for wall-to-wall thematic map quality assessment without sampling, where the test and reference maps feature the same spatial resolution and spatial extent, but whose legends differ and must be harmonized.

show abstract

Quality assessment of classification and cluster maps without ground truth knowledge

Cited by 74 publications

References 35 publications

Assessing the accuracy of land cover change with imperfect ground reference data

Assessing the accuracy of land cover change with imperfect ground reference data

Automatic Spectral Rule-Based Preliminary Mapping of Calibrated Landsat TM and ETM+ Images

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation - Part 1: Theory

Contact Info

Product

Resources

About