Uncertainty-aware generative models for inferring document class prevalence

Keith, Katherine A.; O’Connor, Brendan

doi:10.18653/v1/d18-1487

Cited by 21 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth emphasizing that here we focus on estimating the prevalence using only human labels and assume that we do not have access to the whole unlabeled population. This is in contrast to the body of research on prevalence estimation [6,22], also known as quantification [3,[13][14][15]29] or class prior estimation [34,38], which use supervised learning to train a classifier and make predictions on unlabeled data to infer the prevalence in the population.…”

Section: Prevalence Measurementmentioning

confidence: 99%

Clara

Nguyen

Shi

Ramakrishnan

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Large online services employ thousands of people to label content for applications such as video understanding, natural language processing, and content policy enforcement. While labelers typically reach their decisions by following a well-defined "protocol," humans may still make mistakes. A common countermeasure is to have multiple people review the same content; however, this process is often time-intensive and requires accurate aggregation of potentially noisy decisions. In this paper, we present CLARA (Confidence of Labels and Raters), a system developed and deployed at Facebook for aggregating reviewer decisions and estimating their uncertainty. We perform extensive validations and describe the deployment of CLARA for measuring the base rate of policy violations, quantifying reviewers' performance, and improving their efficiency. In our experiments, we found that CLARA (a) provides an unbiased estimator of violation rates that is robust to changes in reviewer quality, with accurate confidence intervals, (b) provides an accurate assessment of reviewers' performance, and (c) improves efficiency by reducing the number of reviews based on the review certainty, and enables the operational selection of a threshold on the cost/accuracy efficiency frontier. CCS CONCEPTS • Information systems → Crowdsourcing; • Computing methodologies → Latent variable models.

show abstract

Section: Prevalence Measurementmentioning

confidence: 99%

Clara

Nguyen

Shi

Ramakrishnan

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…Post-prediction inference appears across fields and has been recognized as a potential source of error in recent work on prevalence estimation (see for example [20] and [21] in the context of data set shift and [22] in document class prevalence estimation). Here, we focus on developing analytical and bootstrap-based approaches to correct regression estimates, standard errors, and test statistics in inferential regression models using predicted outcomes.…”

Section: Introductionmentioning

confidence: 99%

Post-prediction inference

Wang

McCormick

Leek

2020

Preprint

View full text Add to dashboard Cite

Many modern problems in medicine and public health leverage machine learning methods to predict outcomes based on observable covariates [1,2,3,4]. In an increasingly wide array of settings, these predicted outcomes are used in subsequent statistical analysis, often without accounting for the distinction between observed and predicted outcomes [1,5,6,7,8,9].We call inference with predicted outcomes post-prediction inference. In this paper, we develop methods for correcting statistical inference using outcomes predicted with an arbitrary machine learning method. Rather than trying to derive the correction from the first principles for each machine learning tool, we make the observation that there is typically a low-dimensional and easily modeled representation of the relationship between the observed and predicted outcomes. We build an approach for the post-prediction inference that naturally fits into the standard machine learning framework. We estimate the relationship between the observed and predicted outcomes on the testing set and use that model to correct inference on the validation set and subsequent statistical models. We show our postpi approach can correct bias and improve variance estimation (and thus subsequent statistical inference) with predicted outcome data. To show the broad range of applicability of our approach, we show postpi can improve inference in two totally distinct fields: modeling predicted phenotypes in repurposed gene expression data [10] and modeling predicted causes of death in verbal autopsy data [11]. We have made our method available through an open-source R package: [https://github.com/SiruoWang/postpi]

show abstract

“…Hopkins and King (2010) routinely provided confidence intervals for their estimates "via standard bootstrapping procedures", without commenting much on details of the procedures or on any issues encountered with them. Keith and O'Connor (2018) proposed and compared a number of methods for constructing such confidence intervals. Some of these methods involve Monte-Carlo simulation and some do not.…”

Section: Introductionmentioning

confidence: 99%

“…• Would it be worthwhile to distinguish confidence and prediction intervals for class prevalences and deploy different methods for their estimation? This question is raised against the backdrop that for instance Keith and O'Connor (2018) talked about estimating confidence intervals but in fact constructed prediction intervals which are conceptionally different (Meeker et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

Confidence Intervals for Class Prevalences under Prior Probability Shift

Tasche

2019

MAKE

View full text Add to dashboard Cite

Point estimation of class prevalences in the presence of data set shift has been a popular research topic for more than two decades. Less attention has been paid to the construction of confidence and prediction intervals for estimates of class prevalences. One little considered question is whether or not it is necessary for practical purposes to distinguish confidence and prediction intervals. Another question so far not yet conclusively answered is whether or not the discriminatory power of the classifier or score at the basis of an estimation method matters for the accuracy of the estimates of the class prevalences. This paper presents a simulation study aimed at shedding some light on these and other related questions.

show abstract

Uncertainty-aware generative models for inferring document class prevalence

Cited by 21 publications

References 30 publications

Clara

Clara

Post-prediction inference

Confidence Intervals for Class Prevalences under Prior Probability Shift

Contact Info

Product

Resources

About