Replica analysis of overfitting in generalized linear regression models

Abstract

Nearly all statistical inference methods were developed for the regime where the number N of data samples is much larger than the data dimension p. Inference protocols such as maximum likelihood (ML) or maximum a posteriori probability (MAP) are unreliable if p = O(N), due to overfitting. This limitation has for many disciplines with increasingly high-dimensional data become a serious bottleneck. We recently showed that in Cox regression for time-to-event data the overfitting errors are not just noise but take mostly the form of a bias, and how with the replica method from statistical physics one can model and predict this bias and the noise statistics. Here we extend our approach to arbitrary generalized linear regression models (GLM), with possibly correlated covariates. We analyse overfitting in ML/MAP inference without having to specify data types or regression models, relying only on the GLM form, and derive generic order parameter equations for the case of L2 priors. Second, we derive the probabilistic relationship between true and inferred regression coefficients in GLMs, and show that, for the relevant hyperparameter scaling and correlated covariates, the L2 regularization causes a predictable direction change of the coefficient vector. Our results, illustrated by application to linear, logistic, and Cox regression, enable one to correct ML and MAP inferences in GLMs systematically for overfitting bias, and thus extend their applicability into the hitherto forbidden regime p= O(N).

Entropy Weight Ensemble Framework for Yield Prediction of Winter Wheat Under Different Water Stress Treatments Using Unmanned Aerial Vehicle-Based Multispectral and Thermal Data

“…This may be due to the redundant information generated by the accumulation of features from multiple growth stages. In addition, the excessive dimensionality of input features also poses the risk of overfitting the machine learning model ( Feng et al, 2017 ; Coolen et al, 2020 ). Among the combinations of EMF, the prediction accuracy of C2 was comparable to a combination with the highest prediction accuracy of C4.…”

The ReIMAGINE Multimodal Warehouse: Using Artificial Intelligence for Accurate Risk Stratification of Prostate Cancer

“…The platform supports the deployment of Interpretable Artificial Intelligence (IAI) and Bayesian inference methods for rapid and scalable risk stratification of prostate cancer. These algorithms will include novel findings around overfitting of data ( Coolen et al, 2017 ; Coolen et al, 2020 ) and latent class models ( Rowley et al, 2017 ) which will help us to stratify patients more correctly.…”

“…To address these combined challenges of high data dimensionality, covariate disparity, and latent cohort heterogeneity, we build a data analytics pipeline (based on the libraries underlying the SaddlePoint-Signature and SaddlePoint-Mosaics software packages https://www.saddlepointscience.com/ ) which combine cross-validation protocols, optimisation tools for covariate selection, and modern mathematical techniques with which to “decontaminate” regression outcomes for the effects of overfitting [see e.g. ( Coolen et al, 2017 ; Sheikh and Coolen, 2019 ; Coolen et al, 2020 )], with the use of modality-specific “meta-covariates”. The latter are personalised and optimised modality-specific risk scores (decontaminated for overfitting), which are subsequently used as integrated digital biomarkers that capture the relevant predictive information in each of the data sources.…”

Analisis Performansi Algoritma Linear Regression dengan Generalized Linear Model untuk Prediksi Penjualan pada Usaha Mikra, Kecil, dan Menengah

“…Penelitian yang dilakukan oleh ACC Coolen dengan judul "Replica analysis of overfitting in generalized linear regression models" menunjukkan hasil Derivasi yang hanya bergantung pada bentuk linear tergeneralisasi dari GLM dan saat memilih prior L2. Karena itu replika perhitungannya tidak perlu diulangi untuk setiap contoh model GLM baru; seperti biasa metode replika berfungsi sebagai kendaraan yang relatif tidak menyakitkan dan elegan tetapi kuat untuk sampai pada kumpulan persamaan parameter orde tertutup, bersama dengan rumus mengungkapkan hubungan antara penduga parameter ML / MAP dan benar nilai-nilai parameter ini [13].…”