Estimating the prediction performance of spatial models via spatial k-fold cross validation

Abstract

In machine learning one often assumes the data are independent when evaluating model performance. However, this rarely holds in practise. Geographic information data sets are an example where the data points have stronger dependencies among each other the closer they are geographically. This phenomenon known as spatial autocorrelation (SAC) causes the standard cross validation (CV) methods to produce optimistically biased prediction performance estimates for spatial models, which can result in increased costs and accidents in practical applications. To overcome this problem we propose a modified version of the CV method called spatial k-fold cross validation (SKCV), which provides a useful estimate for model prediction performance without optimistic bias due to SAC. We test SKCV with three real world cases involving open natural data showing that the estimates produced by the ordinary CV are up to 40% more optimistic than those of SKCV. Both regression and classification cases are considered in our experiments. In addition, we will show how the SKCV method can be applied as a criterion for selecting data sampling density for new research area.

The spatial leave-pair-out cross-validation method for reliable AUC estimation of spatial classifiers

“…To estimate a model's prediction performance where the effect of SAC has been reduced, Pohjankukka et al (2014Pohjankukka et al ( , 2017 and Le Rest et al (2014) proposed spatial cross validation (SCV) to be used for this purpose. The idea in SCV is to estimate a model's prediction performance for a test point r units away from the closest known instances.…”

“…This is done by altering the data in the CV procedure, so that a test point will always be at least r units away from the training data. Following Pohjankukka et al (2017), we call this left out area the dead zone. SCV produces a prediction performance estimate of our model as a function of r , i.e.…”

“…)-an assumption violated by spatial data where close instances tend to be more similar than ones distant from each other. Recently, Pohjankukka et al (2014Pohjankukka et al ( , 2017 and Le Rest et al (2014) have proposed spatial CV (SCV) methods for correcting this bias.…”

Embedding road networks and travel time into distance metrics for urban modelling

“…The most sophisticated validation sampling techniques (hold-out and k-fold) assume data in both the test and training sets to be independent of each other. This is an assumption that may be unrealistic with datasets containing SAC, especially if the purpose of the modelling is for interpolation or close proximity extrapolation (Pohjankukka et al 2017). As such, four sampling techniques are considered, three of which consider spatial dependence for comparison (see 'data sampling' in Figure 3):…”

“…(2) spatially stratified 10-fold cross-validation (spatially stratified k-fold crossvalidation [SSKCV]) on the full dataset of 3669; (3) chequerboard holdout on a training set of 1832 properties, with a test set of 1837 properties; (4) spatial k-fold cross-validation (SKCV) (Pohjankukka et al 2017) on samples of the entire dataset, with each sample including 3187 properties AE 135 for each fold.…”