Response to Johnson B.A. Scale Issues Related to the Accuracy Assessment of Land Use/Land Cover Maps Produced Using Multi-Resolution Data: Comments on “The Improvement of Land Cover Classification by Thermal Remote Sensing”. Remote Sens. 2015, 7, 8368–8390

Abstract

Following the suggestion made by Johnson (Johnson B.A., 2015), a polygonbased cross validation (CV) method is compared to the pixel-based CV method to classify different levels of land cover categories using a single-date Landsat 8 image and time series of Landsat TM images. Also, different variants of band combinations, with and without the thermal bands, were considered. The results demonstrate that the inclusion of thermal information into the classification process will improve the classification performance, as was already shown in our original study (Sun and Schulz, 2015). However, it is also demonstrated that the polygon-based CV method produced lower overall accuracy values when compared to the pixel-based CV method. This confirms the argument made by Johnson that a correlation of calibration and validation data due to random sampling of multi-scale data will overestimate the performance of the classifier, and independent polygon-based CV methods have to be applied instead.

We Must all Pay More Attention to Rigor in Accuracy Assessment: Additional Comment to “The Improvement of Land Cover Classification by Thermal Remote Sensing”. Remote Sens. 2015, 7, 8368–8390

“…This doesn't seem to be of concern in the case of [2], because the proportion w i. of validation pixels belonging to class i is close to W i given that the authors took care that the total area of reference polygons of each class was proportional to that in the OBS map. But in the 'polygon cross-validation' of [4], the w i. can differ considerably from their respective W i depending on what particular reference polygons get selected for validation in each run. See [1] for an explanation of how to correctly estimate the proportion of map area that has class i in the map and class j in the reference, for each possible pair (i,j) including (i,i).…”

“…There were some apparent errors also for the grassland class, where some of the reference polygons had clear signs of agricultural practices (not shown). Whenever faulty reference polygons are selected for validation in the corrected method of [4], all pixels inside them count as correctly classified in the confusion matrix, whereas in reality most are wrong (an exception in the faulty polygon of Figure 1 are the few 30-m pixels corresponding to the uppermost corner of the shape, which indeed appears to be mixed forest). I asked the authors if they had used Google Earth to verify the label of reference polygons; they said that only in very few instances where the appearance in the Landsat image was doubtful.…”

“…Intrigued by the rarity of a formal comment on the accuracy assessment of a paper about land cover mapping, a topic in which I am interested (e.g., [5,6]), I read the original paper, comment, and response [2][3][4]. Alas, I came to the conclusion that there were still some issues, and felt that if they were left undiscussed, one could be left with the impression that the accuracy assessment in [4] was correct.…”

“…Alas, I came to the conclusion that there were still some issues, and felt that if they were left undiscussed, one could be left with the impression that the accuracy assessment in [4] was correct. I contacted both the authors and the editors.…”

“…In essence, the authors, who had delineated a number of individual polygons of each class for use as 'ground truth' (446 polygons in total, of 5 ha mean size), were separating reference pixels into training (90%) and validation (10%) in a way that placed most validation pixels adjacent to training pixels, which in the case of the thermal band meant that they had overlapping footprints. In their response to Johnson's comment, Sun and Schulz [4] acknowledged this problem and implemented the solution suggested by Johnson; that is, reserving all the pixels of some of the reference polygons for validation, so that training and validation pixels would be distant from each other. With the corrected method, OA decreased about 5%, but the inclusion of thermal band still improved classification performance.…”