Classification Efficacy Using K-Fold Cross-Validation and Bootstrapping Resampling Techniques on the Example of Mapping Complex Gully Systems

Abstract

The availability of aerial and satellite imageries has greatly reduced the costs and time associated with gully mapping, especially in remote locations. Regardless, accurate identification of gullies from satellite images remains an open issue despite the amount of literature addressing this problem. The main objective of this work was to investigate the performance of support vector machines (SVM) and random forest (RF) algorithms in extracting gullies based on two resampling methods: bootstrapping and k-fold cross-validation (CV). In order to achieve this objective, we used PlanetScope data, acquired during the wet and dry seasons. Using the Normalized Difference Vegetation Index (NDVI) and multispectral bands, we also explored the potential of the PlanetScope image in discriminating gullies from the surrounding land cover. Results revealed that gullies had significantly different (p < 0.001) spectral profiles from any other land cover class regarding all bands of the PlanetScope image, both in the wet and dry seasons. However, NDVI was not efficient in gully discrimination. Based on the overall accuracies, RF’s performance was better with CV, particularly in the dry season, where its performance was up to 4% better than the SVM’s. Nevertheless, class level metrics (omission error: 11.8%; commission error: 19%) showed that SVM combined with CV was more successful in gully extraction in the wet season. On the contrary, RF combined with bootstrapping had relatively low omission (16.4%) and commission errors (10.4%), making it the most efficient algorithm in the dry season. The estimated gully area was 88 ± 14.4 ha in the dry season and 57.2 ± 18.8 ha in the wet season. Based on the standard error (8.2 ha), the wet season was more appropriate in gully identification than the dry season, which had a slightly higher standard error (8.6 ha). For the first time, this study sheds light on the influence of these resampling techniques on the accuracy of satellite-based gully mapping. More importantly, this study provides the basis for further investigations into the accuracy of such resampling techniques, especially when using different satellite images other than the PlanetScope data.

Intersecting views of gully erosion in South Africa

“…Several papers implemented multiple study types ( n = 13), which also had a strong causal factor, processes, and impacts component (41%; Table 2). Methodological approaches ( n = 7) consisted of assessing gully risk (Watson & Ramokgopa, 1997) and evaluating new detection methods related to spectral properties (Makaya, Mutanga, et al, 2019; Mararakanye & Nethengwe, 2012), vegetation indices (Phinzi et al, 2021; Phinzi & Ngetar, 2017; Taruvinga, 2008), and digital elevation models (DEMs) (Olivier et al, 2022). Larger temporal scale studies investigated gully evolution ( n = 4), mainly in arid regions, providing context relating to the influence of connectivity cycles on gully channel migration (Grenfell et al, 2014; Manjoro et al, 2012; Pulley et al, 2018).…”

Machine Learning Application in G.I.S. and Remote Sensing: An Overview

“…An example of cross-validation is K-fold cross-validation (Learn, 2022). The data is divided into K parts, where 1 part is used as a validation dataset and the other remaining as a training dataset (Phinzi, Abriha, & Szabó, 2021). And this process is repeated K times to reduce the biases and produce an effective model(A.…”

Assessment of Forest Cover Changes in Vavuniya District, Sri Lanka: Implications for the Establishment of Subnational Forest Reference Emission Level

“…For example, a previous study from the eastern part of South Africa investigated the accuracy of classification methods in support vector machines and random forests using bootstrapping and k-fold cross-validation methods. The support vector machine with k-fold cross-validation yielded producer and user accuracies above 80% [102]. However, the k-fold cross-validation and bootstrapping methods have many merits and demerits in terms of sampling size, time consumption, linearity, and randomness.…”