Relative Cost Based Model Selection for Sparse High-Dimensional Linear Regression Models

Abstract

In this paper, we propose a novel model selection method named multi-beta-test (MBT) for the sparse high-dimensional linear regression model. The estimation of the correct subset in the linear regression problem is formulated as a series of hypothesis tests where the test statistic is based on the relative least-squares cost of successive parameter models. The performance of MBT is compared to existing model selection methods for high-dimensional parameter space such as extended Bayesian information criterion (EBIC), extended Fisher Information criterion (EFIC), residual ratio thresholding (RRT) and orthogonal matching pursuit (OMP) with a priori knowledge of the sparsity. Simulation results indicate that the performance of MBT in identifying the true support set surpasses that of EBIC, EFIC and RRT in certain regions of the considered parameter settings.

Robust Information Criterion for Model Selection in Sparse High-Dimensional Linear Regression Models

“…The performance of EBIC R is compared with the 'oracle', EBIC, EFIC and MBT. However, the performance comparison with the RRT [18] method is dropped since it behaves quite similar to MBT (see [19] for details). The 'oracle' criterion assumes a priori knowledge of the true cardinality k 0 .…”

“…A similar trend as in EBIC R is observed even in EBIC and EFIC for different choices of γ and c. Hence, to maintain fairness, the following tuning parameter settings are considered for further analysis: ζ = 1 (EBIC R ), c = 1 (EFIC) and γ = 1 (EBIC). For MBT [19], lim N →∞ PCMS → 1 as β → 1. Hence, we choose β = 0.999.…”

“…When γ = 0, EBIC boils down to BIC (19). Moreover, unlike BIC, EBIC is consistent in selecting the true model for p ≫ N cases where p grows exponentially with N .…”

“…However, the performance of CV is quite poor in sample scarce scenarios with large parameter dimensions and even though CV is unbiased, it can have high variance [17]. Recent additions to the list of model selection methods for highdimensional data are residual ratio thresholding (RRT) [18] and multi-beta-test (MBT) [19]. Both are non-information theoretic methods based on hypothesis testing using a test statistic.…”

Model Selection in High-Dimensional Block-Sparse Linear Regression

“…Apart from the methods mentioned above, there are other non-IC based model selection approaches in the HD regime such as the Residual-Ratio-Thresholding (RRT) [9] and the Multi-Beta-Test (MBT) [10], both of which are based on the hypothesis testing framework used along with a greedy variable selection method such as orthogonal matching pursuit (OMP) [11]. Some recent methods also include significance test of the LASSO [12] and knock-off-filters [13].…”