Problematic Situation Analysis and Automatic Recognition for Chinese Online Conversational System

Xiang, Yang; Zhang, Yaoyun; Zhou, Xiaoqiang; Wang, Xiaolong; Yang, Qin

doi:10.3115/v1/w14-6808

Cited by 13 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without using references, in [131], turn-level coherence is annotated for individual bot utterances and a binary classification model is built using the coherence label and a set of hand-crafted features including dialog acts, question types, predicate-argument structure, named entities, and dependency parsing structure. A model is developed in [132] by training on human annotations of problematic turns to automatically recognize such turns using intent and sentiment features. Using the annotations in the WOCHAT datasets [117], several binary classification models are compared in [133] for estimating whether a bot utterance is valid.…”

Section: Dialog System Evaluation Approachesmentioning

confidence: 99%

From captions to visual concepts and back

Fang

Gupta

Iandola

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

1,072

642

View full text Add to dashboard Cite

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our heldout test set, the system captions have equal or better quality 34% of the time.

show abstract

Section: Dialog System Evaluation Approachesmentioning

confidence: 99%

From captions to visual concepts and back

Fang

Gupta

Iandola

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

1,072

642

View full text Add to dashboard Cite

show abstract

“…(Gandhe and Traum, 2016) propose a semi-automatic evaluation metric for dialogue coherence, similar to BLEU and ROUGE, based on 'wizard of Oz' type data. 6 (Xiang et al, 2014) propose a framework to predict utterance-level problematic situations in a dataset of Chinese dialogues using intent and sentiment factors. Finally, (Higashinaka et al, 2014) train a classifier to distinguish user utterances from system-generated utterances using various dialogue features, such as dialogue acts, question types, and predicate-argument structures.…”

Section: Related Workmentioning

confidence: 99%

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Lowe¹,

Noseworthy²,

Serban³

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 1: Long Papers)

282

265

View full text Add to dashboard Cite

Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality. Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (ADEM) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model's predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and systemlevel. We also show that ADEM can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.

show abstract

“…To this end, we have released the data 1 to the public so that researchers in the field can test their ideas for detecting breakdowns. Although there have been approaches to detecting errors in open-domain conversation, the reported accuracies are not that high (Xiang et al, 2014;Higashinaka et al, 2014b). We believe our taxonomy will be helpful for conceptualizing the errors, and the forthcoming challenge will encourage a more detailed analysis of errors in chat-oriented dialogue systems.…”

Section: Summary and Future Workmentioning

confidence: 99%

Towards Taxonomy of Errors in Chat-oriented Dialogue Systems

Higashinaka

Funakoshi

Araki

et al. 2015

Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

This paper presents a taxonomy of errors in chat-oriented dialogue systems. Compared to human-human conversations and task-oriented dialogues, little is known about the errors made in chat-oriented dialogue systems. Through a data collection of chat dialogues and analyses of dialogue breakdowns, we classified errors and created a taxonomy. Although the proposed taxonomy may not be complete, this paper is the first to present a taxonomy of errors in chat-oriented dialogue systems. We also highlight the difficulty in pinpointing errors in such systems.

show abstract

Problematic Situation Analysis and Automatic Recognition for Chinese Online Conversational System

Cited by 13 publications

References 10 publications

From captions to visual concepts and back

From captions to visual concepts and back

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Towards Taxonomy of Errors in Chat-oriented Dialogue Systems

Contact Info

Product

Resources

About