Combining principal component and robust ridge estimators in linear regression model with multicollinearity and outlier

Arum, Kingsley Chinedu; Ugwuowo, Fidelis Ifeanyi

doi:10.1002/cpe.6803

Cited by 9 publications

(11 citation statements)

References 39 publications

(106 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…OLS minimizes

{‖y - Xβ‖}_{2}^{2}

subject to an L2 norm with respect to

bold-italicβ

but fails to give a unique estimate in high dimensional settings when p > n 21 . Another threat to the performance of OLS is multicollinearity, which surfaces as a result of the correlation or linear dependency among the predictors 22–27 . Biased estimators such as the ridge regression estimator, 28 the Liu estimator, 29 modified ridge‐type estimator, 30 the Kibria–Lukman (KL) estimator, 31 robust principle component (PC)‐ridge estimator, 24 JKL estimator, 22 and others were developed to account for multicollinearity problem in linear regression models.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Another threat to the performance of OLS is multicollinearity, which surfaces as a result of the correlation or linear dependency among the predictors 22–27 . Biased estimators such as the ridge regression estimator, 28 the Liu estimator, 29 modified ridge‐type estimator, 30 the Kibria–Lukman (KL) estimator, 31 robust principle component (PC)‐ridge estimator, 24 JKL estimator, 22 and others were developed to account for multicollinearity problem in linear regression models. The ridge estimator 28 minimizes

{‖y - Xβ‖}_{2}^{2}

subject to an L2‐norm of the coefficients and is defined as follows:

{\overset{truê}{bold-italicβ}}_{n}^{k} goodbreak= \arg \min_{β \in ℝ^{p}} ({}, {‖y - Xβ‖}_{2}^{2} goodbreak+ k {‖bold-italicβ‖}_{2}^{2}) goodbreak= {({bold-italicX}^{T} X + k I_{p})}^{- 1} X^{T} bold-italicy bold,

where

k

> 0 is the tuning parameter and

I_{p}

is the identity matrix with … and is better than the OLS estimator.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

See 1 more Smart Citation

Two‐step hybrid modeling for variable selection and estimation: An application to quantitative structure activity relationship study

Oranye,

Ugwuowo,

Arum

2023

Journal of Chemometrics

Self Cite

View full text Add to dashboard Cite

In this study, we developed a simple technique for effective parameter estimation and prediction of the quantitative structure activity relationship studies using a two‐step procedure. The first step is to choose the important molecular descriptors using the random forest regression, and the second step is to optimally predict the biological activity of the selected chemical compounds using the following estimators: ridge regression, jackknife ridge, Liu regression, jackknife Liu, Kibria–Lukman, and jackknife Kibria–Lukman. We conducted a simulation study and a real‐life analysis with a quantitative structure–activity relationship (QSAR) data with 2540 descriptors after preprocessing. The optimal prediction is determined using the cross‐validation error. The estimator with minimum cross‐validation error is considered best. It is obvious that performing jackknife estimation after random forest selection is preferred.

show abstract

“…OLS minimizes

{‖y - Xβ‖}_{2}^{2}

subject to an L2 norm with respect to

bold-italicβ

Section: Theoretical Backgroundmentioning

confidence: 99%

{‖y - Xβ‖}_{2}^{2}

subject to an L2‐norm of the coefficients and is defined as follows:

{\overset{truê}{bold-italicβ}}_{n}^{k} goodbreak= \arg \min_{β \in ℝ^{p}} ({}, {‖y - Xβ‖}_{2}^{2} goodbreak+ k {‖bold-italicβ‖}_{2}^{2}) goodbreak= {({bold-italicX}^{T} X + k I_{p})}^{- 1} X^{T} bold-italicy bold,

where

k

> 0 is the tuning parameter and

I_{p}

is the identity matrix with … and is better than the OLS estimator.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

Two‐step hybrid modeling for variable selection and estimation: An application to quantitative structure activity relationship study

Oranye,

Ugwuowo,

Arum

2023

Journal of Chemometrics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Outliers are data points that differ significantly from other observations and can have a substantial impact on model estimates 8 , 9 . They threatened the efficiency of the OLS estimator 8 – 11 , and it is well-known that robust estimators are preferred when dealing with outliers 12 – 19 . However, both multicollinearity and outliers can exist simultaneously in a model.…”

Section: Introductionmentioning

confidence: 99%

Robust-stein estimator for overcoming outliers and multicollinearity

Lukman

Farghali

Kibria

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Linear regression models with correlated regressors can negatively impact the performance of ordinary least squares estimators. The Stein and ridge estimators have been proposed as alternative techniques to improve estimation accuracy. However, both methods are non-robust to outliers. In previous studies, the M-estimator has been used in combination with the ridge estimator to address both correlated regressors and outliers. In this paper, we introduce the robust Stein estimator to address both issues simultaneously. Our simulation and application results demonstrate that the proposed technique performs favorably compared to existing methods.

show abstract

“…All of the aforementioned techniques behave differently when an outlier is present. It results in a discernible modification of model estimations, including forecasted values, estimated variation, and others [20][21][22]. Additionally, the presence of an outlier observation violates the premise of normality assumptions [20,21].…”

Section: Introductionmentioning

confidence: 99%

“…Additionally, the presence of an outlier observation violates the premise of normality assumptions [20,21]. Knowing that the estimator's breakdown point is extremely low, for example, one outlier figure had a significant impact on TLS's performance [20][21][22]. When the LRM is tainted by extreme values or significant observations, robust approach is a viable alternative to TLS.…”

Section: Introductionmentioning

confidence: 99%

Robust weighted ridge regression based on S – estimator

Fayose,

Ayinde,

Alabi

et al. 2023

Afr. Sci. Rep.

View full text Add to dashboard Cite

Ordinary least squares (OLS) estimator performance is seriously threatened by correlated regressors often called multicollinearity. Multicollinearity is a situation when there is strong relationship between any two exogenous variables. In this case, the ridge estimator offers more reliable estimations when such scenario occurs. Outliers can have significant impact on the estimation of regression parameters leading to biased results. However, both OLS and Ridge estimators are sensitive to outlaying observations (outliers). The outlier – prone dataset in the endogenous variable has been efficiently solved utilizing the Robust M estimator in our recent paper. In recent studies, Robust M estimator has some drawbacks despite its efficient performances with outlier – prone dataset in the dependent (endogenous) variable hence the consideration of the Robust S estimator. The Robust S estimator is less sensitive to outliers in both endogenous and exogenous variables and utilizes standard deviation rather than the median in the case of Robust M estimator to handle outliers in the endogenous variable. The Robust ridge based on the S estimator was well suited to model dataset with multicollinearity and outliers problems in the endogenous variable. Also, it was noted in literature that outliers are one of the causes of heteroscedasticity and non – robust weighted least squares were initially employed to figure out for them. In this study, we introduced proposed Robust S weighted ridge estimator by adding a novel weighting method to address the three problems in a linear regression model. The selected Ridge estimators and Robust S estimator in two weighted versions (True weight (Wo) and suggested new weight (W1)) were combined to respectively develop the Robust S Ridge and Robust S Weighted Ridge estimators. Monte – Carlo simulation experiments were conducted on a linear regression model with three and six exogenous variables exhibiting different degrees of Multicollinearity, with heteroscedasticity structure of powers, size of outlier in endogenous variable and error variances and five levels of sample sizes. The Mean Square Error (MSE) was used as a criterion to evaluate the performances of the new and existing estimators. Simulation findings show categorically that the new approach is preferred over previous approach and hereby recommended.

show abstract

Combining principal component and robust ridge estimators in linear regression model with multicollinearity and outlier

Cited by 9 publications

References 39 publications

Two‐step hybrid modeling for variable selection and estimation: An application to quantitative structure activity relationship study

Two‐step hybrid modeling for variable selection and estimation: An application to quantitative structure activity relationship study

Robust-stein estimator for overcoming outliers and multicollinearity

Robust weighted ridge regression based on S – estimator

Contact Info

Product

Resources

About