Using Machine Learning to Predict Geomorphic Disturbance: The Effects of Sample Size, Sample Prevalence, and Sampling Strategy

Abstract

Advances in data acquisition and statistical methodology have led to growing use of machine‐learning methods to predict geomorphic disturbance events. However, capturing the data required to parameterize these models is challenging because of expense or, more fundamentally, because the phenomenon of interest occurs infrequently. Thus, it is important to understand how the nature of the data used to train predictive models influences their performance. Using a database of cliff failure prediction and associated covariates from Auckland, New Zealand, we assess the performance of seven machine‐learning algorithms under different sampling strategies. Three sampling components are investigated: (i) the number of data points used in model training (sample size), (ii) the prevalence of occurrences (presences) in the data, and (iii) random versus spatial sampling strategy. Across the seven algorithms, small sample sizes can produce models that perform relatively well, especially if the prime concern is identifying key predictors rather than quantifying risk or predicting categorical outcome. Our analyses show that for the same effort (i.e., number of samples), sampling around multiple locations provides better predictions than sampling at just one or a few locations. Predictive performance may be further improved by considering issues such as the nature of what absences actually represent and paying careful attention to decisions about hyperparameter tuning, training‐testing data splits, and threshold optimization. It is well known that big data can inform complex data‐driven modeling, but here we show that careful sampling can facilitate informative event prediction even from small data.

An Automated Python Language-Based Tool for Creating Absence Samples in Groundwater Potential Mapping

“…However, the random sampling method has some drawbacks. First, this method does not pay attention to the distribution pattern of absence samples, and therefore the absence samples generated are sometimes significantly clustered and do not provide overall information on the entire study area [34,38]. Second, absence samples may be very close to presence locations, resulting in confusion in the model and also increasing an error in the final output [39].…”

Investigation and Optimization of the C-ANN Structure in Predicting the Compressive Strength of Foamed Concrete

“…In this study, a thorough investigation of ANN structures has been carried out for predicting the 28 days CS of FC, which is one of the most important mechanical properties of FC. In addition, it is well-known that the accuracy of the given machine learning algorithm greatly depends on the sampling strategy to construct the model [85,86]. results in [25,47].…”

“…In this study, a thorough investigation of ANN structures has been carried out for predicting the 28 days CS of FC, which is one of the most important mechanical properties of FC. In addition, it is well-known that the accuracy of the given machine learning algorithm greatly depends on the sampling strategy to construct the model [85,86]. Therefore, MCS was used in this study to fully analyze the capability of all the C-ANN structures, taking into account such variability of the input space in the training phase of the model.…”

Using Support Vector Machine (SVM) and Ionospheric Total Electron Content (TEC) Data for Solar Flare Predictions

“…Current studies have clarified patterns of spatial sensitivity, however temporal forecasts have remained largely empirical [49], [50]. Most ML techniques achieve overall success rates of 75 − 95% [51]. While this may seem very promising, there are issues which remain with data input quality, potential over fitting and inadequate choice of prediction models, introducing unintentional inclusion of redundant or noise variables, and technical limits to predicting only certain types and sizes of the flare event [52], [53], [54].…”