MIDAS: A Dialog Act Annotation Scheme for Open Domain HumanMachine Spoken Conversations

Yu, Dian; Yu, Zhou

doi:10.18653/v1/2021.eacl-main.94

Cited by 14 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This tagging scheme, while provides more detailed information compared with previous works, only focuses on the system side in a task-oriented setting and may need major modifications when applied to open-domain dialogues. After the initial flourish, recent works come with their own definitions and tagsets for dialog act, tailored for different scenarios or special needs (Budzianowski et al, 2018;Yu and Yu, 2019).…”

Section: Dialog Act In Dialogue Systemsmentioning

confidence: 99%

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Zhao¹,

Li²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation. * Equal contributions. Wanyu participated in building the segment act dataset, while doing her internship with Liwei.

show abstract

Section: Dialog Act In Dialogue Systemsmentioning

confidence: 99%

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Zhao¹,

Li²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As the self-disclosure levels were contextdependent, we included the bot's utterance of the last turn, and the previous user utterance segments of the current turn in the input to classify each user utterance segment. Inspired by (Yu and Yu, 2021)'s method of context representation, we appended the bot's last utterance (bot last turn), the user's utterance prior to the target segment of the same turn (user prev segs), and the target user segmented text (user cur seg) as [CLS] bot last turn : user prev segs [SEP] user cur seg [SEP]. If there was no previous user segment, we then put an EMPTY token in the user prev segs.…”

Section: Classifiermentioning

confidence: 99%

Discovering Chatbot's Self-Disclosure's Impact on User Trust, Affinity, and Recommendation Effectiveness

Liang¹,

Shi²,

Oh³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, chatbots have been empowered to engage in social conversations with humans and have the potential to elicit people to disclose their personal experiences, opinions, and emotions. However, how and to what extent people respond to chabots' self-disclosure remain less known. In this work, we designed a social chatbot with three self-disclosure levels that conducted small talks and provided relevant recommendations to people. 372 MTurk participants were randomized to one of the four groups with different self-disclosure levels to converse with the chatbot on two topics, movies, and COVID-19. We found that people's self-disclosure level was strongly reciprocal to a chatbot's self-disclosure level. Chatbots' self-disclosure also positively impacted engagement and users' perception of the bot and led to a more effective recommendation such that participants enjoyed and agreed more with the recommendations.

show abstract

“…For example, we can enhance the recall of Regexes intent detection by incorporating existing deep learning-based NLU (Natural Language Understanding) models. Specifically, we re-purpose an open-sourced dialog act classification model (Yu and Yu, 2021) to enhance disengagement intent detection: we select 6 out of the 23 supported dialog act labels that are associated with disen-gaged intents, and map each selected dialog act label to the heuristic groups. The dialog act "complaint" is mapped to the heuristic group "complain system repetition";"closing" is mapped to the disengaged intent "request termination"; "hold" to "hesitation";"other_answers" to "unsure answer"; "back-channeling" to "back-channeling", and "neg_answer" to 'negative answer'".…”

Section: Heuristic Functions Implementationmentioning

confidence: 99%

HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Liang

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Open-domain dialog systems have a usercentric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora. Our implementation is available at https:// github.com/Weixin-Liang/HERALD/.

show abstract

MIDAS: A Dialog Act Annotation Scheme for Open Domain HumanMachine Spoken Conversations

Cited by 14 publications

References 20 publications

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Discovering Chatbot's Self-Disclosure's Impact on User Trust, Affinity, and Recommendation Effectiveness

HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Contact Info

Product

Resources

About