Controlling Risk of Web Question Answering

Su, Lixin; Guo, Jiafeng; Fan, Yixing; Lan, Yanyan; Cheng, Xueqi

doi:10.1145/3331184.3331261

Cited by 7 publications

(6 citation statements)

References 36 publications

(72 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few existing works have addressed the conversational search risks in making these decisions, which is a key factor in real-world IR applications. Su et al [34] propose a risk control framework for web question answering. They use an uncertainty model, which estimates the predictive uncertainty, and a decision model, which decides whether to answer the question, to control the risk of their reading comprehension system.…”

Section: Risk Controlmentioning

confidence: 99%

Simulating and Modeling the Risk of Conversational Search

Wang

2022

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

In conversational search, agents can interact with users by asking clarifying questions to increase their chance to find better results. Many recent works and shared tasks in both NLP and IR communities have focused on identifying the need of asking clarifying questions and methodologies of generating them. These works assume asking a clarifying question is a safe alternative to retrieving results. As existing conversational search models are far from perfect, it’s possible and common that they could retrieve/generate bad clarifying questions. Asking too many clarifying questions can also drain user’s patience when the user prefers searching efficiency over correctness. Hence, these models can get backfired and harm users’ search experience because of these risks by asking clarifying questions. In this work, we propose a simulation framework to simulate the risk of asking questions in conversational search and further revise a risk-aware conversational search model to control the risk. We show the model’s robustness and effectiveness through extensive experiments on three conversational datasets, including MSDialog, Ubuntu Dialog Corpus, and Opendialkg in which we compare it with multiple baselines. We show that the risk-control module can work with two different re-ranker models and outperform all the baselines in most of our experiments.

show abstract

Section: Risk Controlmentioning

confidence: 99%

Simulating and Modeling the Risk of Conversational Search

Wang

2022

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…TriviaQA [21] and SearchQA [22]. [23] introduced a risk control framework to manage the uncertainty of deep learning models in Web QA. The authors argue that there are two forms of risks, by returning: (1) wrong answers for answerable questions; and (2) any answers for unanswerable questions.…”

Section: Unanswerable Questionsmentioning

confidence: 99%

“…This is similar to the risks 8 https://cseweb.ucsd.edu/$\sim$jmcauley 9 https://github.com/zswvivi/icdm_pqa 10 The original implementation uses a softmax activation function to compute P (r|q) (and so the probability of all reviews sum up to one); we make a minor modification to the softmax function and use a sigmoid function instead (and so each review produces a valid probability distribution over the positive and negative classes). proposed for Web QA [23], although a crucial difference is that PQA is a ranking problem (output is a list of reviews). The implication is that when deciding the threshold to guarantee a "error rate", we are using a ranking metric as opposed to a classification metric.…”

Section: Rejection Modelmentioning

confidence: 99%

“…The MC models are trained to first detect whether a question is answerable or unanswerable and then find answers to answerable questions. [23] proposed a risk controlling framework to increase the reliability of MC models, where risks are quantified based on the extent of incorrect predictions, for both answerable and unanswerable questions. Different from MC which always has one answer, PQA is a ranking problem where there can be a number of relevant reviews/answers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Less is More: Rejecting Unreliable Reviews for Product Question Answering

Zhang

Lau

et al. 2020

Preprint

View full text Add to dashboard Cite

Promptly and accurately answering questions on products is important for e-commerce applications. Manually answering product questions (e.g. on community question answering platforms) results in slow response and does not scale. Recent studies show that product reviews are a good source for real-time, automatic product question answering (PQA). In the literature, PQA is formulated as a retrieval problem with the goal to search for the most relevant reviews to answer a given product question. In this paper, we focus on the issue of answerability and answer reliability for PQA using reviews. Our investigation is based on the intuition that many questions may not be answerable with a finite set of reviews. When a question is not answerable, a system should return nil answers rather than providing a list of irrelevant reviews, which can have significant negative impact on user experience. Moreover, for answerable questions, only the most relevant reviews that answer the question should be included in the result. We propose a conformal prediction based framework to improve the reliability of PQA systems, where we reject unreliable answers so that the returned results are more concise and accurate at answering the product question, including returning nil answers for unanswerable questions. Experiments on a widely used Amazon dataset show encouraging results of our proposed framework. More broadly, our results demonstrate a novel and effective application of conformal methods to a retrieval task.

show abstract

“…Kamath et al (2020) study calibration in the context of selective answering, i.e., learning when QA models should abstain from answering questions. They show that training a forecaster to predict the model's confidence can perform well when facing a distributional shift Su et al (2019). also investigate selective answering using a probe in the model to determine the model's confidence.…”

mentioning

confidence: 99%

Calibration of Machine Reading Systems at Scale

Dhuliawala¹,

Adolphs²,

Das³

et al. 2022

Preprint

View full text Add to dashboard Cite

In typical machine learning systems, an estimate of the probability of the prediction is used to assess the system's confidence in the prediction. This confidence measure is usually uncalibrated; i.e. the system's confidence in the prediction does not match the true probability of the predicted output. In this paper, we present an investigation into calibrating open setting machine reading systems such as opendomain question answering and claim verification systems. We show that calibrating such complex systems which contain a discrete retrieval and deep reading components is challenging and current calibration techniques fail to scale to these settings. We propose simple extensions to existing calibration approaches that allow us to adapt these callibrators to these settings.Our experimental results reveal that the joint callibration of the retriever and the reader outperforms the reader calibrator by a significant margin. We also show that the callibrator can be useful for selective prediction, e.g., when question answering systems are posed with unanswerable or out-of-the-training distribution questions.

show abstract

Controlling Risk of Web Question Answering

Cited by 7 publications

References 36 publications

Simulating and Modeling the Risk of Conversational Search

Simulating and Modeling the Risk of Conversational Search

Less is More: Rejecting Unreliable Reviews for Product Question Answering

Calibration of Machine Reading Systems at Scale

Contact Info

Product

Resources

About