Adaptive selection from multiple response candidates in example-based dialogue

Mizukami, Masahiro; Kizuki, Hideaki; Nomura, Taishin; Neubig, Graham; Yoshino, Katsumi; Sakti, Sakriani; Toda, Tomoki; Nakamura, Satoshi

doi:10.1109/asru.2015.7404868

Cited by 3 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior research proposed to use simple approaches such as random response selection , corrupting utterances by inserting, substituting and deleting random tokens (Whang et al, 2021), using the mask-and-fill 617 approach (Gupta et al, 2021) for generating adversarial negative examples or collect human-written negative samples (Sato et al, 2020). Previous work also suggest to augment dialog datasets with synthetically generated positive samples (Mizukami et al, 2015;Khayrallah and Sedoc, 2020;Gupta et al, 2019;Sai et al, 2020;Zhang et al, 2020a).…”

Section: Related Workmentioning

confidence: 99%

MERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems

Ghazarian,

Hedayatnia,

Jin

et al. 2023

Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

Automatic Evaluation (AE) and Response Selection (RS) models assign quality scores to various candidate responses and rank them in conversational setups. Prior response ranking research compares various models' performance on synthetically generated test sets. In this work, we investigate the performance of model-based reference-free AE and RS models on our constructed response ranking datasets that mirror real-case scenarios of ranking candidates during inference time. Metrics' unsatisfying performance can be interpreted as their low generalizability over more pragmatic conversational domains such as human-chatbot dialogs. To alleviate this issue we propose a novel RS model called MERCY that simulates human behavior in selecting the best candidate by taking into account distinct candidates concurrently and learns to rank them. In addition, MERCY leverages natural language feedback as another component to help the ranking task by explaining why each candidate response is relevant/irrelevant to the dialog context. These feedbacks are generated by prompting large language models in a few-shot setup. Our experiments show the better performance of MERCY over baselines for the response ranking task in our curated realistic datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

MERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems

Ghazarian,

Hedayatnia,

Jin

et al. 2023

Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

show abstract

“…For training response selection models, typically (Mizukami et al, 2015;Khayrallah and Sedoc, 2020;Gupta et al, 2019;Sai et al, 2020;. Within open-domain dialogs, Gupta et al (2019);Sai et al (2020) augmented the Dai-lyDialog dataset with multiple positive human written responses.…”

Section: Related Workmentioning

confidence: 99%

A Systematic Evaluation of Response Selection for Open Domain Dialogue

Hedayatnia,

Jin,

Liu

et al. 2022

Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

Recent progress on neural approaches for language processing has triggered a resurgence of interest on building intelligent open-domain chatbots. However, even the state-of-the-art neural chatbots cannot produce satisfying responses for every turn in a dialog. A practical solution is to generate multiple response candidates for the same context, and then perform response ranking/selection to determine which candidate is the best. Previous work in response selection typically trains response rankers using synthetic data that is formed from existing dialogs by using a ground truth response as the single appropriate response and constructing inappropriate responses via random selection or using adversarial methods. In this work, we curated a dataset where responses from multiple response generators produced for the same dialog context are manually annotated as appropriate (positive) and inappropriate (negative). We argue that such training data better matches the actual use case examples, enabling the models to learn to rank responses effectively. With this new dataset, we conduct a systematic evaluation of state-of-the-art methods for response selection, and demonstrate that both strategies of using multiple positive candidates and using manually verified hard negative candidates can bring in significant performance improvement in comparison to using the adversarial training data, e.g., increase of 3% and 13% in Recall@1 score, respectively.

show abstract