CAsT-19: A Dataset for Conversational Information Seeking

Dalton, Jeff; Xiong, Chenyan; Kumar, Vaibhav; Callan, Jamie

doi:10.1145/3397271.3401206

Cited by 57 publications

(80 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dataset: The CAsT dataset (Dalton et al, 2020) consists of 30 training topics (9 questions per topic, 269 in total), and 50 test topics (9.6 questions per topic, 478 in total). However, relevance judgments are available only for 20 test topics (173 questions).…”

Section: Experimental Methodologymentioning

confidence: 99%

“…The introduction of the CAsT dataset (Dalton et al, 2020) has brought in a new range of systems which focus on conversational information seeking. The ATeam's run (Dalton et al, 2019) of TREC CAsT 2019 utilises GPT-2 (Radford et al, 2019) to translate questions augmented with previous turns of the conversation into stand-alone questions that are afterwards used to retrieve relevant passages.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Making Information Seeking Easier: An Improved Pipeline for Conversational Search

Kumar¹,

Callan²

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

View full text Add to dashboard Cite

This paper presents a highly effective pipeline for passage retrieval in a conversational search setting. The pipeline comprises of two components: Conversational Term Selection (CTS) and Multi-View Reranking (MVR). CTS is responsible for performing the first-stage of passage retrieval. Given an input question, it uses a BERT-based classifier (trained with weak supervision) to de-contextualize the input by selecting relevant terms from the dialog history. Using the question and the selected terms, it issues a query to a search engine to perform the first-stage of passage retrieval. On the other hand, MVR is responsible for contextualized passage reranking. It first constructs multiple views of the information need embedded within an input question. The views are based on the dialog history and the top documents obtained in the first-stage of retrieval. It then uses each view to rerank passages using BERT (fine-tuned for passage ranking). Finally, MVR performs a fusion over the rankings produced by the individual views. Experiments show that the above combination improves first-state retrieval as well as the overall accuracy in a reranking pipeline. On the key metric of NDCG@3, the proposed combination achieves a relative performance improvement of 14.8% over the state-of-the-art baseline and is also able to surpass the Oracle.

show abstract

Section: Experimental Methodologymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Making Information Seeking Easier: An Improved Pipeline for Conversational Search

Kumar¹,

Callan²

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…We also use our OpenMatch library to reimplement different Neu-IR models and conduct ranking results on several benchmarks, as shown in Table 3. Six datasets, ClueWeb09 [4], Robust04 [18], TREC COVID [39], MS MARCO [27], TREC CAsT-19 [12] and TREC CAsT-20 [11], are used in our experiments. With OpenMatch, we reproduce corresponding results of previous work on different ranking benchmarks.…”

Section: Letormentioning

confidence: 99%

OpenMatch: An Open Source Library for Neu-IR Research

Liu

Zhang

Xiong

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

OpenMatch is a Python-based library that serves for Neural Information Retrieval (Neu-IR) research. It provides self-contained neural and traditional IR modules, making it easy to build customized and higher-capacity IR systems. In order to develop the advantages of Neu-IR models for users, OpenMatch provides implementations of recent neural IR models, complicated experiment instructions, and advanced few-shot training methods. OpenMatch reproduces corresponding ranking results of previous work on widely-used IR benchmarks, liberating users from surplus labor in baseline reimplementation. Our OpenMatch-based solutions conduct top-ranked empirical results on various ranking tasks, such as ad hoc retrieval and conversational retrieval, illustrating the convenience of OpenMatch to facilitate building an effective IR system. The library, experimental methodologies and results of OpenMatch are all publicly available at https://github.com/thunlp/OpenMatch.

show abstract

“…Conversational information seeking has been receiving increasing interest in various studies recently. Many of these studies analyzed real human-human interactions to better understand conversational flow and behavior [6,7]. The goal is often to understand and reflect those behaviors in human-conversational AI interactions.…”

Section: Related Workmentioning

confidence: 99%

Direct Answer Threshold Optimization in Dialogue Systems

Peixeiro

Naji

Charton

2021

Proceedings of the Canadian Conference on Artificial Intelligence

View full text Add to dashboard Cite

The presence of dialogue systems is rising in a wide array of industries. While complex, human-like conversational flow and turn-taking have been the focus of recent research advances, we make the point that direct answers are equally important in humanbot interactions. This is especially true in an information seeking task where prompt, correct answers with minimal back-and-forth are desirable. We define a direct answer as a response given to a user query without requiring further clarifications from the user. To determine whether a direct answer is to be given or not, a threshold is applied to the to the confidence level of the predicted intent; in the case where the confidence is higher than the threshold, the user receives a direct answer. This threshold is often set intuitively or on the basis of a few observations, usually between 50% -75%. In this paper, we propose a method to estimate this threshold based on the intent classification confidence level combined with several intent volumetrics. The goal of our method is to maximize the number of correct direct responses for as many intents as possible in order to minimize user frustration from unnecessary requests for clarifications. Moreover, our method is applicable in the earlier stages of a dialogue system when real interaction logs are scarce. We show that our method improves the accuracy of directly answered queries by 3 to 14% while maximizing the number of accurately answered intents on two dialogue system datasets of 32 and 152 intents.

show abstract

CAsT-19: A Dataset for Conversational Information Seeking

Cited by 57 publications

References 5 publications

Making Information Seeking Easier: An Improved Pipeline for Conversational Search

Making Information Seeking Easier: An Improved Pipeline for Conversational Search

OpenMatch: An Open Source Library for Neu-IR Research

Direct Answer Threshold Optimization in Dialogue Systems

Contact Info

Product

Resources

About