Predictive Inference with Weak Supervision

Abstract

The expense of acquiring labels in large-scale statistical machine learning makes partially and weakly-labeled data attractive, though it is not always apparent how to leverage such data for model fitting or validation. We present a methodology to bridge the gap between partial supervision and validation, developing a conformal prediction framework to provide valid predictive confidence sets-sets that cover a true label with a prescribed probability, independent of the underlying distribution-using weakly labeled data. To do so, we introduce a (necessary) new notion of coverage and predictive validity, then develop several application scenarios, providing efficient algorithms for classification and several large-scale structured prediction problems. We corroborate the hypothesis that the new coverage definition allows for tighter and more informative (but valid) confidence sets through several experiments.

Conformal Prediction is Robust to Label Noise

“…To our knowledge, conformal prediction under label noise has not been previously analyzed. The closest work to ours is that of Cauchois et al (2022) studying conformal prediction with weak supervision, which could be interpreted as a type of noisy label.…”

“…In most supervised classification and regression tasks, one would assume the provided labels reflect the ground truth. In reality, this assumption is often violated; see (Cheng et al, 2022;Xu et al, 2019;Yuan et al, 2018;Lee & Barber, 2022;Cauchois et al, 2022). For example, doctors labeling the same medical image may have different subjective opinions about the diagnosis, leading to variability in the ground truth label itself.…”

Recommendation Systems with Distribution-Free Reliability Guarantees

“…Our approach is based on recent developments in conformal prediction [Vovk et al, 1999, Papadopoulos et al, 2002, Vovk et al, 2005 and distribution-free uncertainty quantification more broadly [Park et al, 2020, Bates et al, 2021a. This line of work provides a formal approach to defining set-valued statistical predictions and it has been applied to various learning tasks, such as distribution estimation [Vovk et al, 2020], causal inference Candès, 2020, Jin et al, 2021], weakly-supervised data [Cauchois et al, 2022], survival analysis , design [Fannjiang et al, 2022], model cascades [Fisch et al, 2020, the few-shot setting , handling dependent data [Chernozhukov et al, 2018, Dunn et al, 2020, and handling or testing distribution shift [Tibshirani et al, 2019, Cauchois et al, 2020, Hu and Lei, 2020, Bates et al, 2021b, Gibbs and Candès, 2021, Vovk, 2021, Podkopaev and Ramdas, 2022. Most closely related to the present work, there have been recent proposals applying conformal prediction to recommender systems.…”