Classification accuracy comparison: Hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

Foody, Giles M.

doi:10.1016/j.rse.2009.03.014

Cited by 198 publications

(153 citation statements)

References 42 publications

Supporting

Mentioning

142

Contrasting

Unclassified

Order By: Relevance

“…Finally, it should be noted that comparative studies need not only focus on the basic difference between values. A range of other scenarios exist, such as testing for equivalence and non-inferiority, and the null hypothesis is not constrained to be one of no difference (Foody, 2009c).…”

Section: Accuracy and Comparisonmentioning

confidence: 99%

An overview of recent remote sensing and GIS based research in ecological informatics

Boyd

Foody

2011

Ecological Informatics

114

View full text Add to dashboard Cite

This article provides an overview of some of the recent research in ecological informatics involving remote sensing and GIS. Attention focuses on a selected range of issues including topics such as the nature of remote sensing data sets, issues of accuracy and uncertainty, data visualization and sharing activities as well as developments in aspects of ecological modeling research. It is shown that considerable advances have been made over recent years and foundations for future research established. BackgroundGeotechnology has been couched as one of the three "mega-technologies" of the 21 st century, along with nanotechnology and biotechnology (Gewin, 2004). On the cusp of the second decade of the 21 st century it is evident that recent developments in the geotechnologies of Geographic Information Science (GIS) and remote sensing have had a substantial impact on ecological research, providing spatial data and associated information to enable the further understanding of ecological systems (Rundell et al., 2009). In a digital society there is sometimes an expectation that information is just a click away and this may apply to spatial information in the domain of ecological informatics. Indeed, it has been in the last five years that developments in and relating to the fields of GIS and remote sensing, such as those seen in internet technologies (e.g., Web 2.0; high performance communication networks; Brunt et al., 2007), sensing technology (e.g., quantum well infrared photodetectors; Krabach, 2000), data standards and interoperability (e.g., OGC® KML 2.2) and spatially explicit modeling (e.g., Osborne et al., 2007), have resulted in the revelation of previously unobservable phenomena and posing of what might be termed second generation of ecological questions. During this time we have also seen more support for the open exchange of data and software and infrastructure for location based services. This paper serves as a review of some of the recent developments in GIS and remote sensing that have been or promises to be of benefit to ecological informatics. The subjects addressed are necessarily selective and limited in scope, drawing on topics of interest to us and, we hope, to you.

show abstract

Section: Accuracy and Comparisonmentioning

confidence: 99%

An overview of recent remote sensing and GIS based research in ecological informatics

Boyd

Foody

2011

Ecological Informatics

114

View full text Add to dashboard Cite

show abstract

“…On the basis of the error table, we wish to achieve the statistical significance of the differences between the proportions using McNemar's test [76] so as to assess two allocated results, which are obtained by two SCDDs, or just given, under a position-specific comparison. Here, McNemar's test is specifically useful for comparing paired proportions derived from two sets of samples.…”

Section: Mcnemar's Testmentioning

confidence: 99%

“…McNemar's test will perform the evaluation of the 100(1 − α)% confidence interval for comparing the difference between two accuracy values based on the differences (D α ) between the proportions. Assuming a normal distribution z α , the general expression of the confidence interval [76] can be expressed as:…”

Section: Mcnemar's Testmentioning

confidence: 99%

“…In this way, the statistical assessment of the differences is carried out to determine if these are significantly different or not [76]. Since three kinds of error matrices are presented in this study, here we name them as CDt (i.e., the classifier-dependent using the test set), CIa (i.e., the classifier-independent using the reference map of the "sa"), and CIb (i.e., the classifier-independent using the reference map of the "sb").…”

Section: Mcnemar's Testmentioning

confidence: 99%

See 1 more Smart Citation

Single-Class Data Descriptors for Mapping Panax notoginseng through P-Learning

Deng

2018

Applied Sciences

View full text Add to dashboard Cite

Machine learning-based remote-sensing techniques have been widely used for the production of specific land cover maps at a fine scale. P-learning is a collection of machine learning techniques for training the class descriptors on the positive samples only. Panax notoginseng is a rare medicinal plant, which also has been a highly regarded traditional Chinese medicine resource in China for hundreds of years. Until now, Panax notoginseng has scarcely been observed and monitored from space. Remote sensing of natural resources provides us new insights into the resource inventory of Chinese materia medica resources, particularly of Panax notoginseng. Generally, land-cover mapping involves focusing on a number of landscape classes. However, sometimes a subset or one of the classes will be the only part of interest. In term of this study, the Panax notoginseng field is the right unit class. Such a situation makes single-class data descriptors (SCDDs) especially significant for specific land-cover interpretation. In this paper, we delineated the application such that a stack of SCDDs were trained for remote-sensing mapping of Panax notoginseng fields through P-learning. We employed and compared SCDDs, i.e., the simple Gaussian target distribution, the robust Gaussian target distribution, the minimum covariance determinant Gaussian, the mixture of Gaussian, the auto-encoder neural network, the k-means clustering, the self-organizing map, the minimum spanning tree, the k-nearest neighbor, the incremental support vector data description, the Parzen density estimator, and the principal component analysis; as well as three ensemble classifiers, i.e., the mean, median, and voting combiners. Experiments demonstrate that most SCDDs could achieve promising classification performance. Furthermore, this work utilized a set of the elaborate samples manually collected at a pixel-level by experts, which was intended to be a benchmark dataset for the future work. The measuring performance of SCDDs gives us challenging insights to define the selection criteria and scoring proof for choosing a fine SCDD in mapping a specific landscape class. With the increment of remotely sensed satellite data of the study area, the spatial distribution of Panax notoginseng could be continuously derived in the local area on the basis of SCDDs.

show abstract

“…Furthermore, one of the most widely promoted means for classification accuracy comparison is through the comparison of kappa coefficients (Foody 2009). With this approach, the statistical significance of the difference between two independent kappa coefficients is calculated by a z-value with z = where k1 and k2 represent respectively the kappa coefficients for two classifications and σ 2 k1 and σ 2 k2 their associated variance (Congalton and Green 1998).…”

Section: Assessment and Comparison For Classification Performancementioning

confidence: 99%

High Spatial Resolution Hyperspectral Mapping for Forest Ecosystem at Tree Species Level

Shen

Sakai

Hoshino

2010

Agricultural Information Research

View full text Add to dashboard Cite

Integrated management of forest ecosystems requires an accurate and all-sided mastery of the forest information, of which forest ecosystem cover especially at tree species level is the most basic and important component. The study investigated and demonstrated the mapping potential of the forest ecosystem at tree species level from high spatial resolution hyperspectral images. The mapping performances of eight conventional classification methods including Maximum Likelihood (ML), Mahalanobis Distance (MaD), Minimum Distance (MD), Parallelepiped (P), Binary Encoding (BE), Spectral Angle Mapper (SAM), Spectral Information Divergence (SID) and Support Vector Machine (SVM), were verified based on two noise treatments (noise fraction and noise removal) and three leaf growth periods (tender leaf period, young leaf period and adult leaf period). It could be confirmed that noise removal obviously contributed to improving the classification agreement and young leaf period was most suitable for mapping forest ecosystem at tree species level from high spatial resolution hyperspectral images. ML, P, BE and SID were not considered appropriate according to good results with overall accuracy and kappa coefficient exceeding 85% and 0.80 respectively. Though MD also produced a very high classification agreement, it could not cover up its poor potential to identify tree species by spectral features. Even if SVML, SVMP, SVMR and SVMS performed the stablest and could generate good results across three periods, the best result was obtained by SAM. Except that the difference was significant between MD and SVMS at the 5% significance level in tender leaf period, the comparative tests did not provide more proof to show the significant difference between the methods considered appropriate.

show abstract

Classification accuracy comparison: Hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

Cited by 198 publications

References 42 publications

An overview of recent remote sensing and GIS based research in ecological informatics

An overview of recent remote sensing and GIS based research in ecological informatics

Single-Class Data Descriptors for Mapping Panax notoginseng through P-Learning

High Spatial Resolution Hyperspectral Mapping for Forest Ecosystem at Tree Species Level

Contact Info

Product

Resources

About