Lip-Reading: Furhat Audio Visual Intelligibility of a Back Projected Animated Face

Abstract

Abstract. Back projecting a computer animated face, onto a three dimensional static physical model of a face, is a promising technology that is gaining ground as a solution to building situated, flexible and human-like robot heads. In this paper, we first briefly describe Furhat, a back projected robot head built for the purpose of multimodal multiparty human-machine interaction, and its benefits over virtual characters and robotic heads; and then motivate the need to investigating the contribution to speech intelligibility Furhat's face offers. We present an audio-visual speech intelligibility experiment, in which 10 subjects listened to short sentences with degraded speech signal. The experiment compares the gain in intelligibility between lip reading a face visualized on a 2D screen compared to a 3D back-projected face and from different viewing angles. The results show that the audio-visual speech intelligibility holds when the avatar is projected onto a static face model (in the case of Furhat), and even, rather surprisingly, exceeds it. This means that despite the movement limitations back projected animated face models bring about; their audio visual speech intelligibility is equal, or even higher, compared to the same models shown on flat displays. At the end of the paper we discuss several hypotheses on how to interpret the results, and motivate future investigations to better explore the characteristics of visual speech perception 3D projected faces.

Impact of iris size and eyelids coupling on the estimation of the gaze direction of a robotic talking head by human viewers

“…They show that robot faces having a human-like physiognomy and, surprisingly, video projected on a flat screen perform equally well. Instead of interposing a transparent grid between the participants and the faces, Al Moubayed et al [15] [16] replicated part of these experiments with a chessboard using a lamp avatar called Furhat. They compared the estimation by human observers of the gaze of Furhat, its virtual model and the video of its performance both displayed on screen.…”

Tutoring Robots

“…6. Snapshots of Furhat 4 in close-ups interaction [34], audio-visual intelligibility of physically three dimensional avatars [35], and effects of head-pose on accuracy of addressee selection [36]. Figure 6 shows some snapshots of the Furhat robot head.…”

Towards rich multimodal behavior in spoken dialogues with embodied agents

“…The solution to build a talking head using the technique used in Furhat is superior in that: 1) Using a three dimensional head allows for situated and multiparty interaction that is not possible to establish accurately with avatars projected on two dimensional surfaces, thanks to its ability to eliminate the socalled Mona Lisa gaze effect [15,16], and as a result make spoken dialogue more fluent [17] and 2) The use of facial animation instead of other mechatronic solutions to build robot heads enables the use of highly advanced and natural dynamics that are not so easily possible with mechanical servos and artificial skin, thanks to the advanced in facial animation techniques [18]. Furhat, in addition to being a platform to implement models of spoken human-human interaction, has become a vehicle to facilitate research on human-robot interaction [19,20,21], such as studying the effects of gaze movements in multiparty turn-taking [17,22], audio-visual intelligibility of physically three dimensional avatars [23], and effects of head-pose on accuracy of addressee selection [21].…”