Performance Prediction for Non-Factoid Question Answering

Hashemi, Helia; Zamani, Hamed; Croft, W. Bruce

doi:10.1145/3341981.3344249

Cited by 22 publications

(12 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While effective, this setting produces point estimates for each candidate in D, removing all uncertainty and confidence estimates, from the predictions. At this point, areas of research such as QPP and cut off prediction try to determine these quantities heuristically via score and document distributions [2,8,18,31,49,50,56]. This task has become increasingly challenging with the changing nature of neural retrieval models as previously established post-retrieval QPP methods are not as effective for neural models [19].…”

Section: Problem Statement and Motivationmentioning

confidence: 99%

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

Cohen

Mitra

Lesota

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction. CCS CONCEPTS• Information systems → Information retrievals; Retrieval models and ranking; • Computer systems organization → Neural networks.

show abstract

Section: Problem Statement and Motivationmentioning

confidence: 99%

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

Cohen

Mitra

Lesota

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Early research in QPP utilizes linguistic information [29], statistical features [15,23,24] in pre-retrieval methods, or analyses clarity [15,16], robustness [7,20,48,54,55], retrieval scores [34,41,44,47,55] for post-retrieval prediction, which further evolves into several effective frameworks [17,20,28,38,40,45,46]. The QPP techniques have also been explored and analyzed in [3,5,6,10,18,21,22,25,35,36,39,42,43,52,53,27]. With the recent development deep learning techniques, NeuralQPP [51] achieves promising results by training a three-components deep network under weak supervision of existing methods.…”

Section: Related Workmentioning

confidence: 99%

“…With the recent development deep learning techniques, NeuralQPP [51] achieves promising results by training a three-components deep network under weak supervision of existing methods. Recently, while NQA-QPP [22] uses BERT to generate contextualized embedding for QPP in non-factoid question answering, BERT-QPP [4] directly applies BERT with pointwise learning in the prediction task, outperforming previous methods on the MS MARCO dev set [30] and TREC Deep Learning track query sets [13,14]. Groupwise Ranking.…”

Section: Related Workmentioning

confidence: 99%

“…Indeed, recent results demonstrate that BERT effectively improves the performance of post-retrieval QPP [4,22]. For instance, training with a large number of sparse-labeled queries and their highest-ranked documents, BERT-QPP [4] examines the effectiveness of BERT on the MS MARCO [30] and TREC DL [13,14] datasets, by pointwise modeling of query-document pairs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Groupwise Query Performance Prediction with BERT

Chen¹,

He²,

Sun³

2022

Preprint

View full text Add to dashboard Cite

While large-scale pre-trained language models like BERT have advanced the state-of-the-art in IR, its application in query performance prediction (QPP) is so far based on pointwise modeling of individual queries. Meanwhile, recent studies suggest that the cross-attention modeling of a group of documents can effectively boost performances for both learning-to-rank algorithms and BERT-based re-ranking. To this end, a BERT-based groupwise QPP model is proposed, in which the ranking contexts of a list of queries are jointly modeled to predict the relative performance of individual queries. Extensive experiments on three standard TREC collections showcase effectiveness of our approach.

show abstract

“…BERT [18] is a large-scale network based on Transformer which is pre-trained for a language modeling task. BERT has recently proven to be effective in a wide range of NLP and IR tasks, including question answering [18], passage re-ranking [32,34], query performance prediction [20], and conversational QA [40]. The coloring in Figure 2 shows shared parameters.…”

Section: End To End Modeling and Trainingmentioning

confidence: 99%

Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search

Hashemi¹,

Zamani²,

Croft³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Asking clarifying questions in response to ambiguous or faceted queries has been recognized as a useful technique for various information retrieval systems, especially conversational search systems with limited bandwidth interfaces. Analyzing and generating clarifying questions have been studied recently but the accurate utilization of user responses to clarifying questions has been relatively less explored. In this paper, we enrich the representations learned by Transformer networks using a novel attention mechanism from external information sources that weights each term in the conversation. We evaluate this Guided Transformer model in a conversational search scenario that includes clarifying questions.In our experiments, we use two separate external sources, including the top retrieved documents and a set of different possible clarifying questions for the query. We implement the proposed representation learning model for two downstream tasks in conversational search; document retrieval and next clarifying question selection. Our experiments use a public dataset for search clarification and demonstrate significant improvements compared to competitive baselines.

show abstract

Performance Prediction for Non-Factoid Question Answering

Cited by 22 publications

References 6 publications

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

Groupwise Query Performance Prediction with BERT

Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search

Contact Info

Product

Resources

About