A handsome set of metrics to measure utterance classification performance in spoken dialog systems

David, Sina; Liscombe, Jackson; Dayanidhi, Krishna; Pieraccini, Roberto

doi:10.3115/1708376.1708428

Cited by 8 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiment design aims to show that there is a statistical correlation between the presence of "Unazuki" nodding and change of a classic quantitative metrics used to measure the quality of conversational agents, called utterance amount for testing H1. These metrics measure the length of dialogue sessions and have been shown to be well related to Caller Experience, a subjective measure describing how well a human interlocutor was treated by the dialog system [30]. The quantitative evaluation based on the utterance amount is illustrated in Fig.…”

Section: Measurements Of Dependent Variablesmentioning

confidence: 99%

See 1 more Smart Citation

Embodiment matters: toward culture-specific robotized counselling

Sakurai

Kurashige

Tsuruta

et al. 2020

J Reliable Intell Environ

View full text Add to dashboard Cite

In this paper, we propose adding the traditional Japanese nodding behavior to the repertoire of social movements to be used in the context of humanrobot interaction. Our approach is motivated by the notion that in many cultures, trust-building can be boosted by small body gestures. We discuss the integration of a robot capable of such movements within CRECA, our context-respectful counseling agent. The frequent nodding called "unazuki" in Japan, often accompanying the "un-un" sound (meaning "I agree") of Japanese onomatopoeia, underlines empathy and embodies unconditioned approval. We argue that "unazuki" creates more empathy and promotes longer conversation between the robotic counsellor and people. We set up an experiment involving 10 subjects to verify these effects. Our quantitative evaluation is based on the classic metrics of utterance, adapted to support the Japanese language. Interactions featuring "unazuki" showed higher value of this metrics. Moreover, subjects assessed the counselling robot's trustworthiness and kindness as "very high" (Likert scale: 5.5 versus 3 -4.5) showing the effect of social gestures in promoting empathetic dialogue to general people including the younger generation. Our findings support the importance of social movements when using robotized agents as a therapeutic tool aimed at improving emotional state and social interactions, with unambiguous evidence that embodiment can have a positive impact that warrants further exploration. The 3D printable design of our robot supports creating culture-specific libraries of social movements, adapting the gestural repertoire to different human cultures.

show abstract

Section: Measurements Of Dependent Variablesmentioning

confidence: 99%

“…Fig. 7 illustrates the estimation of the conversational agent performance based on utterance mount [30]. The corresponding survey's settings are shown in Table 2.…”

Section: Measurements Of Dependent Variablesmentioning

confidence: 99%

Embodiment matters: toward culture-specific robotized counselling

Sakurai

Kurashige

Tsuruta

et al. 2020

J Reliable Intell Environ

View full text Add to dashboard Cite

show abstract

“…In addition, numerous other aspects of the spoken dialog system can be "learned" for a specific task, such as application-specific grammars [64,65], prompt wording [16,45], choice of text-to-speech audio [11], and others. Learning in these areas can certainly improve performance of a spoken dialog system, but is separate from the dialog management task.…”

Section: Related Workmentioning

confidence: 99%

A Case Study of Applying Decision Theory in the Real World

Williams

2012

Decision Theory Models for Applications in Artificial Intelligence

View full text Add to dashboard Cite

Spoken dialog systems present a classic example of planning under uncertainty. Speech recognition errors are ubiquitous and impossible to detect reliably, so the state of the conversation can never be known with certainty. Despite this, the system must choose actions to make progress to a long term goal. As such, decision theory, and in particular partially-observable Markov decision processes (POMDPs), present an attractive approach to building spoken dialog systems. Initial work on “toy” dialog systems validated the benefits of the POMDP approach; however, it also found that straightforward application of POMDPs could not scale to real-world problems. Subsequent work by a number of research teams has scaled up planning and belief monitoring, incorporated high-fidelity user simulations, and married commercial development practices with automatic optimization. Today, statistical dialog systems are being fielded by research labs for public use. This chapter traces the history of POMDP-based spoken dialog systems, and sketches avenues for future work.

show abstract

“…Crowdsourcing is one solution that allows us to overcome this obstacle of obtaining data rapidly for iterative model building and refinement. Crowdsourcing has been used to rapidly and cheaply obtain data for a number of spoken language applications in recent years, such as native (Suendermann‐Oeft, Liscombe, & Pieraccini, ) and nonnative (Evanini, Higgins, & Zechner, ) speech transcription and evaluation of quality of speech synthesizers (Buchholz & Latorre, ; Wolters, Isaac, & Renals, ). Crowdsourcing, and particularly Amazon Mechanical Turk, has also been used for assessing SDSs and for collecting interactions with SDSs.…”

mentioning

confidence: 99%

Bootstrapping Development of a Cloud‐Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data

Ramanarayanan

Suendermann-Oeft

Lange

et al. 2016

ETS Research Report Series

View full text Add to dashboard Cite

We propose a crowdsourcing‐based framework to iteratively and rapidly bootstrap a dialog system from scratch for a new domain. We leverage the open‐source modular HALEF dialog system to deploy dialog applications. We illustrate the usefulness of this framework using four different prototype dialog items with applications in the educational domain and present initial results and insights from this endeavor.

show abstract

A handsome set of metrics to measure utterance classification performance in spoken dialog systems

Cited by 8 publications

References 8 publications

Embodiment matters: toward culture-specific robotized counselling

Embodiment matters: toward culture-specific robotized counselling

A Case Study of Applying Decision Theory in the Real World

Bootstrapping Development of a Cloud‐Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data

Contact Info

Product

Resources

About