Impact of Regressand Stratification in Dataset Shift Caused by Cross-Validation

Abstract

Data that have not been modeled cannot be correctly predicted. Under this assumption, this research studies how k-fold cross-validation can introduce dataset shift in regression problems. This fact implies data distributions in the training and test sets to be different and, therefore, a deterioration of the model performance estimation. Even though the stratification of the output variable is widely used in the field of classification to reduce the impacts of dataset shift induced by cross-validation, its use in regression is not widespread in the literature. This paper analyzes the consequences for dataset shift of including different regressand stratification schemes in cross-validation with regression data. The results obtained show that these allow for creating more similar training and test sets, reducing the presence of dataset shift related to cross-validation. The bias and deviation of the performance estimation results obtained by regression algorithms are improved using the highest amounts of strata, as are the number of cross-validation repetitions necessary to obtain these better results.

A model for eliminating aggregate and specification bias in multivariate fine-scale urban scenarios

“…Even with simulated data, determining the ideal bias threshold and the precise accuracy of these models remains unattainable. The aforementioned constraints have been extensively reviewed by Sáez & Romero-Béjar (2022) and West et al, (2020). The only available option is selecting the model that presents better results through a combined validation approach.…”

An Adaptive, Data-Driven Stacking Ensemble Learning Framework for the Short-Term Forecasting of Renewable Energy Generation

“…Cross-validation is a resampling method used to evaluate machine learning models, and K-fold means that a given data is spilt into K separate folds. One-fold is used to train the model, and K-1 folds are used to validate, and then an individual estimation is obtained by averaging the results of K evaluations [28]. The model can be trained and validated on each fold data, increasing the model's fitness.…”