The acoustics of eye contact

Abstract

An important aspect in short dialogues is attention as is manifested by eye-contact between subjects. In this study we provide a first analysis whether such visual attention is evident in the acoustic properties of a speaker's voice. We thereby introduce the multi-modal GRAS 2 corpus, which was recorded for analysing attention in human-to-human interactions of short daily-life interactions with strangers in public places in Graz, Austria. Recordings of four test subjects equipped with eye tracking glasses, three audio recording devices, and motion sensors are contained in the corpus. We describe how we robustly identify speech segments from the subjects and other people in an unsupervised manner from multi-channel recordings. We then discuss correlations between the acoustics of the voice in these segments and the point of visual attention of the subjects. A significant relation between the acoustic features and the distance between the point of view and the eye region of the dialogue partner is found. Further, we show that automatic classification of binary decision eye-contact vs. no eye-contact from acoustic features alone is feasible with an Unweighted Average Recall of up to 70%.

Explainable Human-centered Traits from Head Motion and Facial Expression Dynamics

“…• For the Excited trait, speech plays a prominent role with a high correlation to continuous or restricted head movement [66]. On the surprising finding of AUs and speech features impacting eye-contact, prior studies [67] have revealed a low-yet-meaningful correlation between eye contact impressions and vocal acoustic features. Friendliness is best characterized by head movement and voice features, showing that the integration of visual and auditory modalities can be crucial in discerning interviewee friendliness [68].…”