Distribution-Free Predictive Inference for Regression

Abstract

We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called rank-one-out conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, in order to adapt to heteroskedascity in the data. Finally, we propose a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference. Accompanying this paper is an R package conformalInference that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.

A comparison of some conformal quantile regression methods

“…As an immediate corollary of Theorem 1, note that it also follows that conformal quantile regression bands have asymptotic conditional coverage, which we define as in [13].…”

Towards safe machine learning for CPS

“…Example 2. Figure 2 shows a toy regression problem, where 40 training samples drawn from a sine function have feature x in [0, 5], and 10 training samples have feature x in (10,15]. However, the testing samples have feature x in (5,10].…”

“…However, the testing samples have feature x in (5,10]. We use a neural network (NN) regressor to fit the data, and as shown, NN does a better job in fitting the sine function in [0, 5] than in (10,15]. Meanwhile, it does a terrible job in extrapolating outside of the training space, i.e., (5, 10].…”

Unpack Local Model Interpretation for GBDT

“…They mainly make changes to feature value and test the chain effect to performance loss of predictions. The loss is then taken as the measure of local importance of feature [11]. This method only relies on the output evaluation and provides an unified way to check feature contribution for black-box models.…”