Causal Decision Making and Causal Effect Estimation Are Not the Same…and Why It Matters

Abstract

Causal decision making (CDM) at scale has become a routine part of business, and increasingly, CDM is based on statistical models and machine learning algorithms. Businesses algorithmically target offers, incentives, and recommendations to affect consumer behavior. Recently, we have seen an acceleration of research related to CDM and causal effect estimation (CEE) using machine-learned models. This article highlights an important perspective: CDM is not the same as CEE, and counterintuitively, accurate CEE is not necessary for accurate CDM. Our experience is that this is not well understood by practitioners or most researchers. Technically, the estimand of interest is different, and this has important implications both for modeling and for the use of statistical models for CDM. We draw on recent research to highlight three implications. (1) We should carefully consider the objective function of the causal machine learning, and if possible, optimize for accurate “treatment assignment” rather than for accurate effect-size estimation. (2) Confounding affects CDM and CEE differently. The upshot here is that for supporting CDM it may be just as good or even better to learn with confounded data as with unconfounded data. (3) Causal statistical modeling may not be necessary at all to support CDM because a proxy target for statistical modeling might do as well or better. This third observation helps to explain at least one broad common CDM practice that seems “wrong” at first blush—the widespread use of noncausal models for targeting interventions. The last two implications are particularly important in practice, as acquiring (unconfounded) data on both “sides” of the counterfactual for modeling can be quite costly and often impracticable. These observations open substantial research ground. We hope to facilitate research in this area by pointing to related articles from multiple contributing fields, most of them written in the last five years.

Measuring the performance of prediction models to personalize treatment choice

“…Moreover, as also noted by Fernández‐Loría and Provost, 17 a model that is optimal in predicting treatment benefit may be suboptimal for making treatment decisions. Thus, we additionally define:…”

“…Likewise, there may be a model that is well-calibrated on average (eg, patients have accurately predicted benefit from 2 to 3%) but may fail to identify which patients among them would gain more benefit from treatment; in this case the model would have good calibration but bad discrimination for benefit. Moreover, as also noted by Fernández-Loría and Provost, 17 a model that is optimal in predicting treatment benefit may be suboptimal for making treatment decisions. Thus, we additionally define:…”

“…This is because in most study designs, we only observe the outcome under a single treatment for each patient, which means that the treatment benefit at the patient level is unobservable 14 . This is related to the “fundamental problem of causal inference.” 15 In the recent years, some approaches have been proposed for assessing a model's capacity to predict treatment benefit 10,11,16,17 ; Schuler et al provide an overview 18 ; in the recent preprint Maas et al described additional methods 19 …”

“…14 This is related to the "fundamental problem of causal inference." 15 In the recent years, some approaches have been proposed for assessing a model's capacity to predict treatment benefit 10,11,16,17 ; Schuler et al provide an overview 18 ; in the recent preprint Maas et al described additional methods. 19 Here, we build on the previous work and present a range of methods for evaluating and comparing models with respect to their ability to predict patient-level treatment effect for a single outcome.…”

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya-Watson Regression

“…Many methods for estimating CATE have been proposed and developed due to importance of the problem in medicine and other applied areas [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. This is only a small part of all publications which are devoted to solving the problem of estimating CATE.…”