A Model Selection Criterion for High-Dimensional Linear Regression

Abstract

Statistical model selection is a great challenge when the number of accessible measurements is much smaller than the dimension of the parameter space. We study the problem of model selection in the context of subset selection for highdimensional linear regressions. Accordingly, we propose a new model selection criterion with the Fisher information that leads to the selection of a parsimonious model from all the combinatorial models up to some maximum level of sparsity. We analyze the performance of our criterion as the number of measurements grows to infinity, as well as when the noise variance tends to zero. In each case, we prove that our proposed criterion gives the true model with a probability approaching one. Additionally, we devise a computationally affordable algorithm to conduct model selection with the proposed criterion in practice. Interestingly, as a side product, our algorithm can provide the ideal regularization parameter for the Lasso estimator such that Lasso selects the true variables. Finally, numerical simulations are included to support our theoretical findings.

Model selection with covariance matching based non-negative lasso

“…Proof of Theorem 2. We would like to highlight that the main steps that should be taken to prove Theorem 2 in this paper are similar to those of Theorem 2 in 320 [11]; thus, to condense this proof, we adopt the presentation in [11,Theorem 2].…”

“…However, since the dimension of the parameter space N is much larger than l, the classical model selection methods, like the Bayesian information criterion (BIC), perform poorly and tend to overestimate the size of the support [24,25]. Specifically, BIC is consistent if N = O(l 1/2 ) [25,26].…”

“…where the expectation is taken with respect to p(z; p I |H I ). In (11), c > 0 is a constant that controls the penalty level in EFIC and its appropriate choice is discussed in Theorem 2.…”

“…In this paper, we provide deterministic conditions under which the support estimate of our method is guaranteed to match the true support. Further, to tune the corresponding regularization parameter in the Lasso estimator, we use the extended Fisher information criterion (EFIC) [11]. We prove that minimizing EFIC yields the true support with probability one as the number of rows in the corresponding design matrix grows to infinity.…”

Relative Cost Based Model Selection for Sparse High-Dimensional Linear Regression Models

“…To overcome some of the issues of EBIC a novel model selection criterion named extended Fisher information criterion (EFIC) was proposed that takes into account the problem of generic model selection for high-dimensional parame-ter vectors [10,11]. It is formulated on the works of EBIC and Fisher information based model selection criteria.…”