Liu regression after random forest for prediction and modeling in high dimension

Abstract

In the modern era, using advanced technology, we have access to data with many features, and therefore, feature engineering has become a vital task in data analysis. One of the challenges in model estimation is to combat multicollinearity in high‐dimensional data problems where the number of features ( p) exceeds the number of samples ()n. We propose a novel, yet simple, strategy to estimate the regression parameters in a high‐dimensional regime in the presence of multicollinearity. The proposed approach enjoys the good properties of the random forest and the simple structure of a class of linear unified estimators. We give a fast and straightforward algorithm to estimate the regression coefficients when p>n and multicollinearity exist. Numerical investigation reveals the superior performance of the method in test mean squared error. The technique is also applied to melting chemical data, where we conducted an estimation among 4885 features and discussed advantages.

Structure and stability characteristics of zonal soil aggregates in the Three Rivers Source of the Qinghai‐Tibetan Plateau

“…The goodness of fit function (goof) was used for cross‐validation to obtain the determination coefficient ( R 2 ), consistency coefficient (CC), mean error (ME), and root mean square error (RMSE) as evaluation indicators for model prediction. The formulae are as follows (Arashi et al., 2022; Lindner et al., 2015): $$$$\begin{equation*}{R}^{\mathrm{2}}{\mathrm{\ = \ }}\mathop \sum \limits_{i{\mathrm{\ = \ 1}}}^N {\left\{ {p{\mathrm{(}}{x}_i{\mathrm{)\ }} - \ \bar{p}} \right\}}^{\mathrm{2}}{\mathrm{/}}\mathop \sum \limits_{{\mathrm{i}}\ {\mathrm{ = \ 1}}}^N {\left\{ {v{\mathrm{(}}{x}_i{\mathrm{)\ }} - \ \bar{p}} \right\}}^{\mathrm{2}}\end{equation*}$$$$$$$$\begin{equation*}{\mathrm{CC\ = \ }}\frac{{{\mathrm{2}} \cdot R \cdot {{{\sigma}}}_v \cdot {{{\sigma}}}_p}}{{{{{\sigma}}}_p^{\mathrm{2}}{\mathrm{\ + \ }}{{{\sigma}}}_v^{\mathrm{2}}{\mathrm{\ + \ }}{{\left( {\bar{v}\ - \ \bar{p}} \right)}}^{\mathrm{2}}}}\end{equation*}$$$$…”

Two‐step hybrid modeling for variable selection and estimation: An application to quantitative structure activity relationship study

“…[32][33][34][35] Random forests can be used to naturally rank the importance of variables in a regression or classification problem, and it is implemented in the R package randomForest or python library Scikit-learn (Sklearn). 36,37 Recently, Arashi et al 21 selected important variables with the use of random forests. The highdimensional data were reduced to low-dimensional data, and finally, the low-dimensional data were estimated using the Liu estimator.…”

“…where the response variable y i is a n  1 vector, the predictors x i ℝ nÂp , i ¼ 1, …, n, is a known n  p matrix of predictors such that p > n, the regression coefficient β is a p  1 vector, and ϵ i , is the random error vector assumed to be normally distributed with mean 0 and variance σ 2 ℝ þ , i ¼ 1, …, n: In low dimensional settings when p < n, β is popularly estimated using the least squares estimator (OLS). OLS minimizes y À Xβ k k 2 2 subject to an L2 norm with respect to β but fails to give a unique estimate in high dimensional settings when p > n. 21 Another threat to the performance of OLS is multicollinearity, which surfaces as a result of the correlation or linear dependency among the predictors. [22][23][24][25][26][27] Biased estimators such as the ridge regression estimator, 28 the Liu estimator, 29 modified ridge-type estimator, 30 the Kibria-Lukman (KL) estimator, 31 robust principle component (PC)-ridge estimator, 24 JKL estimator, 22 and others were developed to account for multicollinearity problem in linear regression models.…”

“…However, recently they have been adopted as post estimation techniques with good performance. 21 The penalized estimation techniques are examples of variable selection methods that select important variables and shrink the irrelevant ones to zero. Much attention has been given to study different penalized estimators.…”

“…They claimed that the hybrid estimators retain the advantages of the combined methods and dampen their drawbacks. [18][19][20][21] Recently, Arashi et al 21 adopted the random forest regression for descriptor selection and employed the ridge or the Liu estimator for prediction. A recent study shows that the jackknife Kibria-Lukman (JKL) estimator outperforms the ridge or the Liu estimator when there is multicollinearity.…”