Improving Propensity Score Estimators' Robustness to Model Misspecification Using Super Learner

Abstract

The consistency of propensity score (PS) estimators relies on correct specification of the PS model. The PS is frequently estimated using main-effects logistic regression. However, the underlying model assumptions may not hold. Machine learning methods provide an alternative nonparametric approach to PS estimation. In this simulation study, we evaluated the benefit of using Super Learner (SL) for PS estimation. We created 1,000 simulated data sets (n = 500) under 4 different scenarios characterized by various degrees of deviance from the usual main-term logistic regression model for the true PS. We estimated the average treatment effect using PS matching and inverse probability of treatment weighting. The estimators' performance was evaluated in terms of PS prediction accuracy, covariate balance achieved, bias, standard error, coverage, and mean squared error. All methods exhibited adequate overall balancing properties, but in the case of model misspecification, SL performed better for highly unbalanced variables. The SL-based estimators were associated with the smallest bias in cases of severe model misspecification. Our results suggest that use of SL to estimate the PS can improve covariate balance and reduce bias in a meaningful manner in cases of serious model misspecification for treatment assignment.

Chasing Balance and Other Recommendations for Improving Nonparametric Propensity Score Models

“…For many of the available approaches, it may be possible to improve performance if more care is taken in how the methods are fine-tuned to select a solution. For example, use of the super learning method which simultaneously runs multiple machine learning methods (including GBM) to estimate propensity scores is currently fine-tuned to select as optimal the combination of machine learners that yields the best prediction (van der Laan 2014, Pirracchio, Petersen, and van der Laan 2015). It may be feasible to improve the already high performance of this method by fine-tuning it so it selects as optimal the combination of machine-learners that yields the best balance.…”

“…In turn, the popularity of propensity scores has given rise to great methodological interest on how best to estimate them. Methods considered have included parametric methods such as logistic regression with or without explicit controls for covariate balance, machine learning methods such as generalized boosted models (GBM), random forests (RF), Bayesian adaptive regression trees (BART), super learning, high dimensional propensity score (hd-PS) methodology, and entropy balancing (van der Laan 2014, Breiman 2001, Hill 2011, Imai and Ratkovic 2014, Liaw and Wiener 2002, McCaffrey, Ridgeway, and Morral 2004b, Pirracchio, Petersen, and van der Laan 2015, Hainmueller 2012). …”

Effect of cricoid pressure on laryngeal view during prehospital tracheal intubation: a propensity-based analysis

“…This was indeed the case in the present sample. Finally, PS model specification is of paramount importance both in terms of the variables included in the model11 and the functional form of the relationship between treatment allocation and the explanatory variables 22. After matching, the best PS model is then the one that offers the best balance across groups.…”

Optimizing variance-bias trade-off in the TWANG package for estimation of propensity scores

“…These methods eliminate reliance on a simple parametric logistic regression model and do not require the researcher to determine which pre-treatment covariates and their respective interactions should be included in the model. It has been shown that the resulting weights from these approaches yield more precise treatment effect estimates and lower mean squared error than traditional logistic regression methods (Harder et al, 2010; Lee et al, 2010; Pirracchio et al, 2015). …”