We Must all Pay More Attention to Rigor in Accuracy Assessment: Additional Comment to “The Improvement of Land Cover Classification by Thermal Remote Sensing”. Remote Sens. 2015, 7, 8368–8390

Abstract

Despite recent calls for statistically robust and transparent accuracy assessment [1], further attention to rigor is still needed. Here I take the opportunity of a disputed accuracy assessment recently published in Remote Sensing [2-4] to highlight some issues regarding sampling design, response design and analysis that I often find as a reviewer, and that I too have neglected in the past, among them: (i) use of a sampling design that is purposive instead of probability based; (ii) use of suboptimal label allocation procedures for reference data; and (iii) incorrect computation of accuracy estimates and lack of reporting of confidence intervals. Following the good practices in [1] will enhance the scientific credibility of our discipline.

Mapping the Dabus Wetlands, Ethiopia, Using Random Forest Classification of Landsat, PALSAR and Topographic Data

“…In addition, as a result of the Dabus Marsh arcuate configuration in the landscape, the spatial distribution of the reference data shows a highly-clustered pattern with expected spatial autocorrelation, which was not measured. Such clustering of reference sample locations can contribute to classification accuracy inflation [80,108]; however, this could not be avoided. Finally, the field surveys represented less than 5-10% of the total area, and were mainly concentrated in the northwestern portion of the Dabus Wetlands.…”

Remote sensing of forest pest damage: a review and lessons learned from a Canadian perspective

“…This deficiency could be addressed by a requirement that "the producer must defend in a statistically valid manner that the overall accuracy of the map exceeds 70% with a 95% confidence", which implicitly dictates the sampling effort required to achieve that confidence. While recommendations on good practices for assessing the accuracy of remote sensing maps exist (e.g., Olofsson et al 2014), in terms of practice there is opportunity for improvement (Castilla 2016). Notwithstanding, accuracy requirements could be less stringent for applications where users only need a broad indication of damage, but this should be reflected in the data quality section of the accompanying metadata.…”

Integrating remote sensing and demography for more efficient and effective assessment of changing mountain forest distribution

“…The benefit of good training and validation data and its importance for robust accuracy assessment has been well discussed elsewhere (Castilla, 2016;Olofsson et al, 2013Olofsson et al, , 2014. However, of the studies highlighted here, only seven (Allen and Walsh, 1996;Bharti et al, 2012;Dinca et al, 2017;Hill et al, 2007;Luo and Dai, 2013;Mihai et al, 2017;Resler et al, 2004) provide a quantitative accuracy assessment of the classification produced, either through a traditional confusion table with percent accuracy or through regression as in Hill et al (2007).…”

“…When monitoring inaccessible areas of mountain ranges, a robust field data set is required to reduce subjectivity when training classification algorithms and to independently assess the accuracy of distribution maps. The importance of accuracy assessment has been highlighted previously (Bharti et al, 2012;Castilla, 2016;Olofsson et al, 2013Olofsson et al, , 2014. However, low incorporation of field data and a quantitative accuracy assessment is a persistent problem in the literature (Table 1).…”