Robust Information Criterion for Model Selection in Sparse High-Dimensional Linear Regression Models

Abstract

Model selection in linear regression models is a major challenge when dealing with high-dimensional data where the number of available measurements (sample size) is much smaller than the dimension of the parameter space. Traditional methods for model selection such as Akaike information criterion, Bayesian information criterion (BIC) and minimum description length are heavily prone to overfitting in the high-dimensional setting. In this regard, extended BIC (EBIC), which is an extended version of the original BIC and extended Fisher information criterion (EFIC), which is a combination of EBIC and Fisher information criterion, are consistent estimators of the true model as the number of measurements grows very large. However, EBIC is not consistent in high signal-to-noise-ratio (SNR) scenarios where the sample size is fixed and EFIC is not invariant to data scaling resulting in unstable behaviour. In this paper, we propose a new form of the EBIC criterion called EBIC-Robust, which is invariant to data scaling and consistent in both large sample size and high-SNR scenarios. Analytical proofs are presented to guarantee its consistency. Simulation results indicate that the performance of EBIC-Robust is quite superior to that of both EBIC and EFIC.

Model Selection in High-Dimensional Block-Sparse Linear Regression

“…Hence, the oracle provides the upper bound on the maximum achievable PCMS for any given setting. The tuning parameters chosen are α = 0.01 for GRRT (as mentioned in [31]) and ζ = 1 (EBIC R ) [7], [8].…”

“…EBIC R is derived under the Bayesian framework of model selection, which starts with deriving the maximum a-posteriori (MAP) criterion and ending with the final EBIC R after suitable modifications and reasonable assumptions. We follow similar steps as in [7], [8], but incorporate the multi-measurement and block structure into it. Let us denote the prior pdf of the parameter vector θ I as p(θ I |H I ), the marginal of y as p(y|H I ) and the prior probability of the model with support I as Pr(H I ).…”

“…First, we perform normalization of FI under both large-N and high-SNR assumption. For this we factorize the ln FI term in a similar manner as performed in [7], [8], [39]…”

“…However, classical methods do not fare well in HD scenarios and quite often lead to overfitted models with false parameters. In the present era, popular IC-based methods for model selection in the HD setting include extended Bayesian IC (EBIC) [5], extended Fisher IC (EFIC) [6], and extended BIC-Robust (EBIC R ) [7], [8].…”

Model selection in multivariate adaptive regressions splines (MARS) using alternative information criteria

“…Since criteria such as AIC and Bayesian information criterion (BIC) have a tendency to overfit in model selection in high-dimensional data; EBIC (extended BIC), EBIC-Robust and EFIC (it is formed by combining extended fisher information criterion and EBIC) have been proposed instead of these criteria. The results showed that the EBIC-Robust criterion performed better in model selection compared to the EFIC and EBIC criteria [ 13 ].…”