Sensitivity analysis of individual treatment effects: A robust conformal inference approach

Abstract

We propose a model-free framework for sensitivity analysis of individual treatment effects (ITEs), building upon ideas from conformal inference. For any unit, our procedure reports the Γ-value, a number which quantifies the minimum strength of confounding needed to explain away the evidence for ITE. Our approach rests on the reliable predictive inference of counterfactuals and ITEs in situations where the training data are confounded. Under the marginal sensitivity model of [Z. Tan, J. Am. Stat. Assoc. 101, 1619-1637 (2006)], we characterize the shift between the distribution of the observations and that of the counterfactuals. We first develop a general method for predictive inference of test samples from a shifted distribution; we then leverage this to construct covariate-dependent prediction sets for counterfactuals. No matter the value of the shift, these prediction sets (resp. approximately) achieve marginal coverage if the propensity score is known exactly (resp. estimated). We describe a distinct procedure also attaining coverage, however, conditional on the training data. In the latter case, we prove a sharpness result showing that for certain classes of prediction problems, the prediction intervals cannot possibly be tightened. We verify the validity and performance of the methods via simulation studies and apply them to analyze real datasets.

Using Machine Learning to Individualize Treatment Effect Estimation: Challenges and Opportunities

“…In such cases, it might be possible to produce instead a range of estimates for the CATE that are consistent with prespecified sensitivity models using recent advances in ML for creating prediction intervals under hidden confounding. [91][92][93][94] On the other hand, even if everything important is measured in the data, there also needs to be enough of it. Although it is not possible to make blanket assessments of exact sample sizes needed for different methods, the amount of data that are needed to produce good estimates generally increases with the number of covariates and the flexibility of the ML model.…”

“…If important confounders are likely to be unmeasured, point estimation of effects is not possible unless further assumptions are made, and standard CATE estimators will output biased estimates. In such cases, it might be possible to produce instead a range of estimates for the CATE that are consistent with prespecified sensitivity models using recent advances in ML for creating prediction intervals under hidden confounding 91–94 . On the other hand, even if everything important is measured in the data, there also needs to be enough of it .…”

Causal Dependence Plots for Interpretable Machine Learning

“…Future work can apply existing methods for uncertainty to show, for example, confidence bands around the plotted dependencies. This can be done for example with conformal prediction (Chernozhukov et al, 2021;Lei & Candès, 2021), including the case of sensitivity analysis (Jin et al, 2023;Yin et al, 2022).…”

TISSUE: uncertainty-calibrated prediction of single-cell spatial transcriptomics improves downstream analyses

“…Traditionally, conformal inference proceeds by fitting a machine learning model on labeled training data, evaluating the model predictions on a small amount of labeled calibration data to build calibrated uncertainties, and then deploying the model to unlabeled test data to obtain both the predicted labels and their uncertainty. Conformal inference has been used to quantify uncertainty of region segmentations in tissue image analysis [67], measure confidence of drug discovery predictions [6], and understand the robustness of clinical treatment effects [26]. To extend the traditional conformal inference framework to spatial gene expression prediction, we make several key modifications to build well-calibrated uncertainties in TISSUE (see Methods).…”