Lasso and Group Lasso with Categorical Predictors: Impact of Coding Strategy on Variable Selection and Prediction

Abstract

Machine learning methods are being increasingly adopted in psychological research. Lasso performs variable selection and regularization, and is particularly appealing to psychology researchers because of its connection to linear regression. Researchers conflate properties of linear regression with properties of lasso; however, we demonstrate that this is not the case for models with categorical predictors. Specifically, the coding strategy used for categorical predictors impacts lasso’s performance but not linear regression. Group lasso is an alternative to lasso for models with categorical predictors. We demonstrate the inconsistency of lasso and group lasso models using a real data set: lasso performs different variable selection and has different prediction accuracy depending on the coding strategy, and group lasso performs consistent variable selection but has different prediction accuracy. Additionally, group lasso may include many predictors when very few are needed, leading to overfitting. Using Monte Carlo simulation, we show that categorical variables with one group mean differing from all others (one dominant group) are more likely to be included in the model by group lasso than lasso, leading to overfitting. This effect is strongest when the mean difference is large and there are many categories. Researchers primarily focus on the similarity between linear regression and lasso, but pay little attention to their different properties. This project demonstrates that when using lasso and group lasso, the effect of coding strategies should be considered. We conclude with recommended solutions to this issue and future directions of exploration to improve implementation of machine learning approaches in psychological science.

Examining individual differences in personality trait changes after negative life events

“…Furthermore, multilevel lasso estimation requires all variables in the model to have a mean of zero and a standard deviation of one. As this has also to apply for categorical variables and time, parameter interpretation may be less intuitive compared to standard REML-estimation (Huang & Montoya, 2020). Hence, standardized fixed effects are better extracted from a standard REML-estimation in which categorical variables and time can be coded as desired to improve interpretation (as done in our exploratory analyses below).…”

How to apply variable selection machine learning algorithms with multiply imputed data: A missing discussion.

“…For this tutorial, we used coordinate descent, the estimation procedure of the glmnet package in R (Friedman et al, 2020). Before describing the steps of creating a LASSO model, we highlight a debate about how to best represent the categories of a categorical predictor variable with more than two categories (Huang & Montoya, 2020). One option is to overparameterize the model so that a reference group is not entered into the model.…”

Examining Individual Differences in Personality Trait Changes After Negative Life Events

“…Furthermore, multilevel lasso estimation requires all variables in the model to have a mean of zero and a standard deviation of one. As this has also to apply for categorical variables and time, parameter interpretation may be less intuitive compared to standard REML-estimation (Huang & Montoya, 2020).…”