Testing conditional independence in supervised learning algorithms

Abstract

We propose the conditional predictive impact (CPI), a consistent and unbiased estimator of the association between one or several features and a given outcome, conditional on a reduced feature set. Building on the knockoff framework of Candès et al. (J R Stat Soc Ser B 80:551–577, 2018), we develop a novel testing procedure that works in conjunction with any valid knockoff sampler, supervised learning algorithm, and loss function. The CPI can be efficiently computed for high-dimensional data without any sparsity constraints. We demonstrate convergence criteria for the CPI and develop statistical inference procedures for evaluating its magnitude, significance, and precision. These tests aid in feature and model selection, extending traditional frequentist and Bayesian techniques to general supervised learning tasks. The CPI may also be applied in causal discovery to identify underlying multivariate graph structures. We test our method using various algorithms, including linear regression, neural networks, random forests, and support vector machines. Empirical results show that the CPI compares favorably to alternative variable importance measures and other nonparametric tests of conditional independence on a diverse array of real and synthetic datasets. Simulations confirm that our inference procedures successfully control Type I error with competitive power in a range of settings. Our method has been implemented in an package, , which can be downloaded from https://github.com/dswatson/cpi.

Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process

“…Model-agnostic PFI confidence intervals that are similar to ours are proposed by Watson and Wright (2019); Williamson et al (2019Williamson et al ( , 2020. We additionally correct for variance underestimation arising from resampling (Nadeau and Bengio, 2003) and relate the estimators to the proposed ground truth PFI.…”

“…This means that the marginal PFI breaks the association between the feature(s) X S and the target Y , but also between X S and all other features X C . For the conditional PFI (cPFI) (Molnar et al, 2020;Watson and Wright, 2019;Hooker and Mentch, 2019;Candès et al, 2018), the expectation is taken over the distribution P X S |X C •P X C Y , so that XS follows the conditional distribution of X S given X C but is still independent of Y . The interpretation of the conditional PFI of a feature is therefore also conditional on all features that are correlated with the feature of interest.…”

“…S can be a permutation of the original vector x S . The conditional PFI requires a conditional sampling mechanism for the feature, such as subgroups (Molnar et al, 2020) or knockoffs (Candès et al, 2018;Watson and Wright, 2019). The estimation of P F I requires unseen data, so that the loss estimates deliver unbiased results (Zheng and van der Laan, 2011; Chernozhukov et al, 2018).…”

“…; Fisher A model f deviates from f due to model bias and variance. Similarly P D and P F I estimates deviate from their ground truth versions DGP-PD and DGP-PFI due to bias, variance, and Monte Carlo integration (MC) Watson and Wright, 2019;Apley and Zhu, 2020). have been proposed.…”

Grouped Feature Importance and Combined Features Effect Plot

“…Hence, interpretations in these regions might be misleading. To avoid this problem, alternatives based on conditional distributions or refitting have been suggested (e.g., Strobl et al, 2008;Nicodemus et al, 2010;Hooker and Mentch, 2019;Watson and Wright, 2019;Molnar et al, 2020). Although the conditional PFI provides a solution to this problem, the interpretation of the score changes.…”