Scoring Interactional Aspects of Human-Machine Dialog for Language Learning and Assessment using Text Features

Abstract

While there has been much work in the language learning and assessment literature on human and automated scoring of essays and short constructed responses, there is little to no work examining text features for scoring of dialog data, particularly interactional aspects thereof, to assess conversational proficiency over and above constructed response skills. Our work bridges this gap by investigating both human and automated approaches towards scoring human-machine text dialog in the context of a real-world language learning application. We collected conversational data of human learners interacting with a cloud-based standards-compliant dialog system, triple-scored these data along multiple dimensions of conversational proficiency, and then analyzed the performance trends. We further examined two different approaches to automated scoring of such data and show that these approaches are able to perform at or above par with human agreement for a majority of dimensions of the scoring rubric.

Towards Continuous Estimation of Dissatisfaction in Spoken Dialog

“…Casual observation suggests that in conversation people are often not shy about indicating, moment by moment, how they feel about things, both in terms of making progress towards their goal and in terms of how happy they are with the contributions and behavior of their interlocutor. To date, however, predictive modeling of quality at the level of turns has been rarely attempted, and has focused mostly on interaction quality and conver-sational proficiency, and in only a few dialog genres, both for human-machine and human-human dialogs (Ultes and Minker, 2014;Ultes et al, 2017a;Lykartsis et al, 2018;Bodigutla et al, 2019;Stoyanchev et al, 2019;Spirina et al, 2016;Ramanarayanan et al, 2019;Ando et al, 2020;Katada et al, 2020). In this work we attempt turn-level quality estimation in human-human dialogs in a new genre: short calls to an unknown merchant to make an appointment or arrange a simple transaction.…”

Exploring Recurrent, Memory and Attention Based Architectures for Scoring Interactional Aspects of Human-Machine Text Dialog

“…This paper aims to expand on the analysis presented in Ramanarayanan et al (2019) more comprehensively along two directions. First, we also investigate constructs of text dialog scoring rubric pertaining to topic development along with those pertaining to interaction, aiming to understand, for the first time, how various feature-engineering and model-engineering methods perform on a broader range of scoring dimensions.…”