Distribution-Free Prediction Sets for Two-Layer Hierarchical Models

Dunn, Robin; Wasserman, Larry; Ramdas, Aaditya

doi:10.1080/01621459.2022.2060112

Cited by 8 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the results of Section 5 could be easily repurposed to construct conformal prediction sets for regression or multi-class classification tasks that achieve valid coverage over subsets of individual test cases with certain unique attributes. In those contexts, our work may lead to a useful alternative framework for dealing with problems of uncertainty estimation under algorithmic fairness constraints (Romano et al, 2020a) or stratified sampling mechanisms (Dunn et al, 2022;Park et al, 2022).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Conformal Frequency Estimation with Sketched Data under Relaxed Exchangeability

Sesia¹,

Favaro²,

Dobriban³

2022

Preprint

View full text Add to dashboard Cite

A flexible method is developed to construct a confidence interval for the frequency of a queried object in a very large data set, based on a much smaller sketch of the data. The approach requires no knowledge of the data distribution or of the details of the sketching algorithm; instead, it constructs provably valid frequentist confidence intervals for random queries using a conformal inference approach. After achieving marginal coverage for random queries under the assumption of data exchangeability, the proposed method is extended to provide stronger inferences accounting for possibly heterogeneous frequencies of different random queries, redundant queries, and distribution shifts. While the presented methods are broadly applicable, this paper focuses on use cases involving the count-min sketch algorithm and a non-linear variation thereof, to facilitate comparison to prior work. In particular, the developed methods are compared empirically to frequentist and Bayesian alternatives, through simulations and experiments with data sets of SARS-CoV-2 DNA sequences and classic English literature.

show abstract

Section: Discussionmentioning

confidence: 99%

“…To achieve ( 16) with any value of M , we follow an approach inspired by Dunn et al (2022). Randomly partition the calibration data into G = n 0 /M multisets Z cal g , for g ∈ [G].…”

Section: Construction Of Confidence Intervals With Unique Coveragementioning

confidence: 99%

Conformal Frequency Estimation with Sketched Data under Relaxed Exchangeability

Sesia¹,

Favaro²,

Dobriban³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The validity of a weighted conformal prediction procedure for covariate shift problems was established in Tibshirani et al [62]. Other settings in which conformal prediction has been studied include functional data [41], random effects models [19], and ranking [12].…”

Section: Conformal Predictionmentioning

confidence: 99%

Conformal Prediction for Network-Assisted Regression

Lunde¹,

Levina²,

Zhu³

2023

Preprint

View full text Add to dashboard Cite

An important problem in network analysis is predicting a node attribute using both network covariates, such as graph embedding coordinates or local subgraph counts, and conventional node covariates, such as demographic characteristics. While standard regression methods that make use of both types of covariates may be used for prediction, statistical inference is complicated by the fact that the nodal summary statistics are often dependent in complex ways. We show that under a mild joint exchangeability assumption, a network analog of conformal prediction achieves finite sample validity for a wide range of network covariates. We also show that a form of asymptotic conditional validity is achievable. The methods are illustrated on both simulated networks and a citation network dataset.

show abstract

“…Prediction sets have a rich statistical history dating back to Wilks (1941), Wald (1943), Scheffe and Tukey (1945), and Tukey (1947Tukey ( , 1948. There is an large body of work on constructing prediction sets with coverage guarantees under various assumptions (see, e.g., Bates et al, 2021;Chernozhukov et al, 2018;Dunn et al, 2018;Lei and Wasserman, 2014;Lei et al, 2013Lei et al, , 2015Lei et al, , 2018aPark et al, 2020Park et al, , 2021Sadinle et al, 2019;Kaur et al, 2022;Qiu et al, 2022;Li et al, 2022;Sesia et al, 2022). Among these, one of the best-known methods is conformal prediction (CP) (see, e.g., Vovk et al, 1999;Papadopoulos et al, 2002;Vovk et al, 2022;Chernozhukov et al, 2018;Dunn et al, 2018;Lei and Wasserman, 2014;Lei et al, 2013Lei et al, , 2018a.…”

Section: Related Workmentioning

confidence: 99%

Joint Coverage Regions: Simultaneous Confidence and Prediction Sets

Dobriban¹,

Lin²

2023

Preprint

View full text Add to dashboard Cite

We introduce Joint Coverage Regions (JCRs), which unify confidence intervals and prediction regions in frequentist statistics. Specifically, joint coverage regions aim to cover a pair formed by an unknown fixed parameter (such as the mean of a distribution), and an unobserved random datapoint (such as the outcomes associated to a new test datapoint). The first corresponds to a confidence component, while the second corresponds to a prediction part. In particular, our notion unifies classical statistical methods such as the Wald confidence interval with distribution-free prediction methods such as conformal prediction. We show how to construct finite-sample valid JCRs when a conditional pivot is available; under the same conditions where exact finite-sample confidence and prediction sets are known to exist. We further develop efficient JCR algorithms, including split-data versions by introducing adequate sets to reduce the cost of repeated computation. We illustrate the use of JCRs in statistical problems such as constructing efficient prediction sets when the parameter space is structured.

show abstract

Distribution-Free Prediction Sets for Two-Layer Hierarchical Models

Cited by 8 publications

References 26 publications

Conformal Frequency Estimation with Sketched Data under Relaxed Exchangeability

Conformal Frequency Estimation with Sketched Data under Relaxed Exchangeability

Conformal Prediction for Network-Assisted Regression

Joint Coverage Regions: Simultaneous Confidence and Prediction Sets

Contact Info

Product

Resources

About