Double Machine Learning based Program Evaluation under Unconfoundedness

Abstract

This paper consolidates recent methodological developments based on DoubleMachine Learning (DML) with a focus on program evaluation under unconfoundedness. DML based methods leverage flexible prediction methods to control for confounding in the estimation of (i) standard average effects, (ii) different forms of heterogeneous effects, and (iii) optimal treatment assignment rules. We emphasize that these estimators build all on the same doubly robust score, which allows to utilize computational synergies. An evaluation of multiple programs of the Swiss Active Labor Market Policy shows how DML based methods enable a comprehensive policy analysis. However, we find evidence that estimates of individualized heterogeneous effects can become unstable.

Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance

“…The extreme values of the predicted CATEs are mainly caused by the propensity scores which are close to the {0, 1} bounds. Similar issues of the DR-learner due to extreme propensity scores have also been documented in the simulation experiments of Knaus et al (2021) as well as in the empirical application of Knaus (2020). The second observation is that for the DR-learner we clearly see how the theoretical arguments of sample-splitting and cross-fitting translate into the finite sample properties of the estimator.…”

“…In this paper, we focus on the above-listed meta-learners in order to provide a contrast between less and more complex algorithms for the estimation of causal effects. Moreover, these particular meta-learners have been extensively studied theoretically as well as applied in various empirical settings, including economics (Knaus, 2020;Jacob, 2021;Sallin, 2021;Valente, 2022), public policy (Kristjanpoller, Michell, & Minutolo, 2021;Shah, Kreif, & Jones, 2021), marketing (Gubela, Lessmann, & Jaroszewicz, 2020;Gubela & Lessmann, 2021), medicine (Lu, Sadiq, Feaster, & Ishwaran, 2018;Duan, Rajpurkar, Laird, Ng, & Basu, 2019) or sports (Goller, 2021). Some further examples of meta-learners proposed in the literature consist of the U-learner and Y-learner (Stadie, Kunzel, Vemuri, & Sekhon, 2018), or more recently the IF-learner (Curth, Alaa, & van der Schaar, 2020) and RA-learner (Curth & van der Schaar, 2021), which are, however, beyond the scope of this paper.…”

“…However, common to all versions in the literature is the estimation of the doubly robust score 26 by machine learning methods in the first stage also known as Double Machine Learning (Chernozhukov et al, 2018). For a comprehensive overview of the CATE estimators building on the doubly robust score see Knaus (2020). The specific algorithm for the DR-learner is then defined as follows:…”

“…Additionally, another reason why we do not use the Lasso and linear learners as such is due to a substantial increase in variance as they are prone to outliers as documented in the simulation studies of Jacob (2020) as well as Knaus et al (2021). Furthermore, Random Forests are a popular choice as a base-learner in empirical studies using meta-learners too (Duan et al, 2019;Knaus, 2020). Lastly, from the practical standpoint there is a vast variety of fast and reliable software implementations of Random Forests which makes it easy to use for practitioners.…”

“…According to the simulation evidence, we advice to employ the cross-fitting version of the DR-learner for the CATE estimation in settings with rather balanced treatment assignment and when large sample is available. Recently, Knaus (2020) proposed the normalized DR-learner, that addresses the problem of unstable CATE predictions due to extreme propensity scores which might be a viable option for smaller sample sizes and settings with unbalanced treatment shares. Lastly, the simulation evidence suggests that the R-learner is in comparison to the DR-learner less prone to unstable performance due to extreme propensity scores.…”