Evaluating Dialogs based on Grice’s Maxims

Jwalapuram, Prathyusha

doi:10.26615/issn.1314-9156.2017_003

Cited by 5 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach is different as RUQ is a diagnostic metric. Jwalapuram (2017) propose a Gricean dialog evaluation where humans rate performance on a Likert scale for each category. Qwaider et al (2017) consider the QUANTITY, RELATION, and MANNER maxims for ranking community question answers.…”

Section: Related Workmentioning

confidence: 99%

“…This mimics a setting where a single researcher would like to do model selection, but we do not consider system evaluation, and leave that to future work. Jwalapuram (2017) propose a Gricean human evaluation dialog. The evaluator is asked to rate performance on a Likert scale for each category.…”

Section: Limitationsmentioning

confidence: 99%

“…One linguistic framework for analyzing conversations is Grice's Cooperative Principle (1975), which consists of Maxims of Conversation that function as guidelines for effective communication. Grice considered conversations between humans, but there has also been some exploration in NLP (Bernsen et al, 1996;Harabagiu et al, 1996;Qwaider et al, 2017;Jwalapuram, 2017).…”

Section: Introductionmentioning

confidence: 99%

“…One theoretical framework for analyzing conversations is Grice's Maxims of Conversation (1975). Grice originally analyzed conversations between humans, but there has been some exploration in the context of evaluating dialog systems as well (Qwaider et al, 2017;Jwalapuram, 2017).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Measuring the ‘I don’t know’ Problem through the Lens of Gricean Quantity

Khayrallah¹,

Sedoc²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

We consider the intrinsic evaluation of neural generative dialog models through the lens of Grice's Maxims of Conversation (1975). Based on the maxim of Quantity (be informative), we propose Relative Utterance Quantity (RUQ) to diagnose the 'I don't know' problem, in which a dialog system produces generic responses. The linguistically motivated RUQ diagnostic compares the model score of a generic response to that of the reference response. We find that for reasonable baseline models, 'I don't know' is preferred over the reference the majority of the time, but this can be reduced to less than 5% with hyperparameter tuning. RUQ allows for the direct analysis of the 'I don't know' problem, which has been addressed but not analyzed by prior work.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Limitationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Measuring the ‘I don’t know’ Problem through the Lens of Gricean Quantity

Khayrallah¹,

Sedoc²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

Towards User-Centric Text-to-Text Generation: A Survey

Yang

Flek

2021

Text, Speech, and Dialogue

View full text Add to dashboard Cite

Evaluation of effectiveness in conversations between humans and chatbots using parallel convolutional neural networks with multiple temporal resolutions

Escobar-Grisales

Vásquez-Correa

Orozco-Arroyave

2023

Multimed Tools Appl

View full text Add to dashboard Cite

Chatbots enable the automation of several components in customer service and allow the support of multiple users. Despite their multiple advantages, due to the large amount of conversations generated by a chatbot, it is difficult to determine whether customer requests are well-addressed. For practical reasons, chatbot’s effectiveness is evaluated manually based upon a small sample (randomly chosen) of conversations or through self-reported user satisfaction. This procedure does not guarantee the correct evaluation of the service because the sample is generally not large enough and self-reports might be influenced by different external factors not directly associated to the chatbot’s functioning. This study proposes a methodology for automatic evaluation of chatbot effectiveness in real production environments. The analysis considers convolutional neural networks adapted for natural language processing, using two parallel convolutional layers to evaluate questions and answers independently. The proposed model also incorporates filters to extract features with multiple temporal resolution. This methodology is tested upon real conversations of chatbots that provide service to two different companies. The results are compared to baseline models based on classical techniques with different pre-trained word embedding models. According to our results, the proposed approach provides accuracies between 78.95% and 80.18%, which outperforms the best result of the baseline models by 2.9%.

show abstract

Evaluating Dialogs based on Grice’s Maxims

Cited by 5 publications

References 14 publications

Measuring the ‘I don’t know’ Problem through the Lens of Gricean Quantity

Measuring the ‘I don’t know’ Problem through the Lens of Gricean Quantity

Towards User-Centric Text-to-Text Generation: A Survey

Evaluation of effectiveness in conversations between humans and chatbots using parallel convolutional neural networks with multiple temporal resolutions

Contact Info

Product

Resources

About