Detecting certainness in spoken tutorial dialogues

Abstract

What role does affect play in spoken tutorial systems and is it automatically detectable? We investigated the classification of student certainness in a corpus collected for ITSPOKE, a speech-enabled Intelligent Tutorial System (ITS). Our study suggests that tutors respond to indications of student uncertainty differently from student certainty. Results of machine learning experiments indicate that acoustic-prosodic features can distinguish student certainness from other student states. A combination of acoustic-prosodic features extracted at two levels of intonational analysis -breath groups and turns -achieves 76.42% classification accuracy, a 15.8% relative improvement over baseline performance. Our results suggest that student certainness can be automatically detected and utilized to create better spoke dialog ITSs.

Recognizing Uncertainty in Speech

“…They listened to the utterances in a random order and had no knowledge of the target words, the questions for the transit items, or the instructions that the speakers were given. The average interannotator agreement (Kappa) was 0.45, which is on par with past work in emotion detection [7,8]. We refer to the mean of the five listeners' ratings for an utterance as the "perceived level of certainty.…”

“…In Section 5, we describe a basic classifier that uses prosodic features computed at the utterance level to classify how certain a person is perceived with an accuracy of 69%. The performance of the basic classifier compares well to prior work [8]. We go beyond prior work by showing improvement over a nonprosodic baseline.…”

“…An intriguing possibility is to use information gleaned from perceived level of certainty to more accurately model the self-reported level. This idea bears promise especially given the potential, pursued by ourselves (Section 5) and others [8], of inferring the perceived level of certainty directly from prosodic information. We pursue this idea in this section, showing that a kind of triage on the perceived level of certainty can improve self-report predictions.…”

“…This definition is reasonable because we are looking for prosodic cues-we want our system to hear whatever it is that humans hear. Not surprisingly, this is the definition that has been assumed in previous work on classifying level of certainty [8]. However, in applications such as spoken tutoring systems [9] and second language learning systems [3], we would like to know how certain speakers actually are-their internal level of certainty, in addition to how certain they are perceived to be.…”

“…On tasks ranging from detecting frustration [5] to detecting flirtation [6], prosody has been shown to convey information about a speaker's emotional and mental state [7] and about their social intentions. Our work builds upon this, as well as a small body of work on identifying prosodic cues to level of certainty [1] and classifying a speaker's certainty [8]. The intended application of such work is for dialogue systems to appropriately respond to a speaker based on their level of certainty as exposed in their prosody, for example, by altering the content of system responses [9] or by altering the emotional coloring of system responses [10].…”