Audiovisual prosody and feeling of knowing

Abstract

This paper describes two experiments on the role of audiovisual prosody for signalling and detecting meta-cognitive information in question answering. The Wrst study consists of an experiment, in which participants are asked factual questions in a conversational setting, while they are being Wlmed. Statistical analyses bring to light that the speakers' Feeling of Knowing (FOK) is cued by a number of visual and verbal properties. It appears that answers tend to have a higher number of marked auditory and visual cues, including divergences from the neutral facial expression, when the FOK score is low, while the reverse is true for non-answers. The second study is a perception experiment, in which a selection of the utterances from the Wrst study is presented to participants in one of three conditions: vision only, sound only, or vision + sound. Results reveal that human observers can reliably distinguish high FOK responses from low FOK responses in all three conditions, but that answers are easier than non-answers, and that a bimodal presentation of the stimuli is easier than the unimodal counterparts. 

Language in dialogue: when confederates might be hazardous to your data

“…Unknown to the subject, the confederate could hear the subject's "privileged" instruction in order to be able to precisely time the delivery of her (supposedly unrelated) critical instruction; he or she also knew whether or not that instruction would be ambiguous to the subject. The confederate therefore could have behaved somewhat differently across conditions (e.g., perhaps giving subjects less time to recover from the distractor in the ambiguous than in the nonambiguous condition; listeners have, after all, been shown to be quite sensitive to the latency before a speaker's utterance; Brennan & Williams, 1995;Swerts & Krahmer, 2005). Keysar et al (1998) addressed this possibility post hoc by comparing the latencies from the end of the distractor instruction to the onset of the critical instruction for the experimental (ambiguous) and baseline (unambiguous) conditions.…”

Mimebot—Investigating the Expressibility of Non-Verbal Communication Across Agent Embodiments

“…We considered the following inner states important to be able to display non-verbally: certainty, a entiveness, engagement, and positive and negative a ect. e rst three states were chosen as they are important for conversational dynamics and have strong audiovisual features [27], [22], and the la er two for increasing the social bonding [6]. e two a ective states were divided into scales according to the cross-diagonal of the circumplex model of emotion [24], one ranging from sad to joyful and the other ranging Uncertain-certain How many pills did I take today |One in the morning two in the a ernoon Una entive-a entive Wake me up at 9 am |mmm-hm |And remind me to go shopping |uh-huh | We need something for Kate |ok Not engaged-engaged Take a right a er the church |mmm-hm |keep going until the roundabout |mmm-hm … Relaxed-frustrated Can we have another round |Sorry the bar closed at one Sad-joyful Do you remember the meeting with Jill last week?…”

“…We primarily target states important for human/robot communication such as displaying levels of certainty [27] and a entiveness [22] but also a ective states important for social bonding and empathy [6]. We collected three data sets where the actor speci cally was asked to use a) only head and face (Face-only), b) only body (Body-only) and c) face and body (Face-and-body) to express a variety of inner states.…”

Pursuing and demonstrating understanding in dialogue

“…This is another indirect kind of grounding. When people speak in problematic situations, their utterances reveal their uncertainty in recognizable ways [Brennan and Willams, 1995, Swerts and Krahmer, 2005, Stone and Oh, 2008. People are simply slower to respond in cases of uncertainty.…”