Private Prediction Sets

Angelopoulos, Anastasios N.; Bates, Stephen; Zrnic, Tijana; Jordan, Michael I.

doi:10.1162/99608f92.16c71dad

Cited by 21 publications

(36 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these scenarios, one would rather use only model predictions that are more likely to be reliable. GP would detect “distribution shifts”, where new datapoints are very different from the training set by assigning high uncertainty to its predictions, while a deep learning model may “fail silently”, generating low-accuracy predictions without providing any obvious sign of failure. ,, Uncertainty can also be used for Bayesian optimization/active learning, where the model assists the practitioner in selecting which datapoints to experimentally evaluate next. − Several techniques for estimating uncertainty in deep learning have been introduced in the literature. − As we will show here, however, the uncertainty estimates they provide are much more poorly calibrated than those provided by a GP.…”

Section: Introductionmentioning

confidence: 99%

Linear-Scaling Kernels for Protein Sequences and Small Molecules Outperform Deep Learning While Providing Uncertainty Quantitation and Improved Interpretability

Parkinson

Wang

2023

J. Chem. Inf. Model.

View full text Add to dashboard Cite

Gaussian process (GP) is a Bayesian model which provides several advantages for regression tasks in machine learning such as reliable quantitation of uncertainty and improved interpretability. Their adoption has been precluded by their excessive computational cost and by the difficulty in adapting them for analyzing sequences (e.g., amino acid sequences) and graphs (e.g., small molecules). In this study, we introduce a group of random feature-approximated kernels for sequences and graphs that exhibit linear scaling with both the size of the training set and the size of the sequences or graphs. We incorporate these new kernels into our new Python library for GP regression, xGPR, and develop an efficient and scalable algorithm for fitting GPs equipped with these kernels to large datasets. We compare the performance of xGPR on 17 different benchmarks with both standard and state-of-the-art deep learning models and find that GP regression achieves highly competitive accuracy for these tasks while providing with well-calibrated uncertainty quantitation and improved interpretability. Finally, in a simple experiment, we illustrate how xGPR may be used as part of an active learning strategy to engineer a protein with a desired property in an automated way without human intervention.

show abstract

Section: Introductionmentioning

confidence: 99%

Linear-Scaling Kernels for Protein Sequences and Small Molecules Outperform Deep Learning While Providing Uncertainty Quantitation and Improved Interpretability

Parkinson

Wang

2023

J. Chem. Inf. Model.

View full text Add to dashboard Cite

show abstract

“…In addition, we compared the number of labeled examples needed to reject a null hypothesis at level 1−α=95% with high probability. See ( 5 ) for a Python package implementing prediction-powered inference, which contains code for reproducing the experiments, and ( 6 ) for the data used in the experiments.…”

Section: Application Of Prediction-powered Inference To Real Datasetsmentioning

confidence: 99%

“…Data and materials availability: The data and code are available in the accompanying Python package ( 5 ) and data repository ( 6 ). All other data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials.…”

Section: Acknowledgmentsmentioning

confidence: 99%

Prediction-powered inference

Angelopoulos,

Bates,

Fannjiang

et al. 2023

Science

View full text Add to dashboard Cite

Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients without making any assumptions about the machine-learning algorithm that supplies the predictions. Furthermore, more accurate predictions translate to smaller confidence intervals. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning. The benefits of prediction-powered inference were demonstrated with datasets from proteomics, astronomy, genomics, remote sensing, census analysis, and ecology.

show abstract

“…With scores for all targets, it is possible to calibrate predictions and in fact use this calibration data for statistically rigorous conformal prediction to generate prediction sets, which contain the true target with some degree of confidence. 21 However, conformal prediction will not be included in the following experiments.…”

Section: Experimental Approachmentioning

confidence: 99%

SAMPLE with a side of MSTAR: extending SAMPLE with outliers and target variants from MSTAR

Lewis

Ashby

Zelnio

2023

Algorithms for Synthetic Aperture Radar Imagery XXX

View full text Add to dashboard Cite

Deep learning is a technology applied to a host of problems in the decade since its introduction. Of particular interest for both defense and civil applications is the technology of automatic target recognition, which is a subset of visual detection and classification. However, these classification algorithms must be robust to out-of-library confusers and able to generalize across a variety of target types. In this paper, we augment the existing Synthetic and Measured Paired Labeled Experiment dataset of synthetic aperture radar images with the remainder of the public MSTAR dataset and define a set of experiments to encourage the development of traits beyond simple classification accuracy for target recognition algorithms.

show abstract

Private Prediction Sets

Cited by 21 publications

References 63 publications

Linear-Scaling Kernels for Protein Sequences and Small Molecules Outperform Deep Learning While Providing Uncertainty Quantitation and Improved Interpretability

Linear-Scaling Kernels for Protein Sequences and Small Molecules Outperform Deep Learning While Providing Uncertainty Quantitation and Improved Interpretability

Prediction-powered inference

SAMPLE with a side of MSTAR: extending SAMPLE with outliers and target variants from MSTAR

Contact Info

Product

Resources

About