Read my lips: speech distortions in musical lyrics can be overcome (slightly) by facial information

Abstract

Understanding the lyrics of many contemporary songs is difficult, and an earlier study [Hidalgo-Barnes, M., Massaro, D.W., 2007. Read my lips: an animated face helps communicate musical lyrics. Psychomusicology 19,[3][4][5][6][7][8][9][10][11][12] showed a benefit for lyrics recognition when seeing a computer-animated talking head (Baldi Ò ) mouthing the lyrics along with hearing the singer. However, the contribution of visual information was relatively small compared to what is usually found for speech. In the current experiments, our goal was to determine why the face appears to contribute less when aligned with sung lyrics than when aligned with normal speech presented in noise. The first experiment compared the contribution of the talking head with the originally sung lyrics versus the case when it was aligned with the Festival text-to-speech synthesis (TtS) spoken at the original duration of the song's lyrics. A small and similar influence of the face was found in both conditions. In the three experiments, we compared the presence of the face when the durations of the TtS were equated with the duration of the original musical lyrics to the case when the lyrics were read with typical TtS durations and this speech embedded in noise. The results indicated that the unusual temporally distorted durations of musical lyrics decreases the contribution of the visible speech from the face.

Seeing a singer helps comprehension of the song’s lyrics

“…38% benefit in Jesse, Vrignaud, Cohen, & Massaro, 2000/2001. As Massaro and Jesse (2009) showed, the ample and atypical temporal distortions imposed on the lyrics by the melody of this particular song significantly attenuated the con-…”

“…Synthetic visual speech animations aided the comprehension of temporally undistorted spoken lyrics: When regularly spoken lyrics were heard and seen, an audiovisual improvement of up to 40% was found (Massaro & Jesse, 2009). The weak audiovisual benefit previously observed for singing was thus probably not due to the quality of the synthetic speaker, but due rather to the visible speech distortions when the face was aligned with the sung lyrics.…”

“…That is, the provided visual information was speech, not singing. Visible singing differs, however, from visible speaking, because it alters articulatory gestures (for a review, see Massaro & Jesse, 2009). For example, lip and jaw openings vary in singing, primarily as a function of pitch (e.g., Sundberg, 1982;Thompson & Russo, 2007).…”

“…Our previous experiments involved the alignment of a computer-animated face with the music and lyrics of a contemporary rock song (Hidalgo-Barnes & Massaro, 2007;Massaro & Jesse, 2009). Although the 7% audiovisual benefit obtained for the comprehension of the sung lyrics was a statistically significant improvement (Hidalgo-Barnes & Massaro, 2007), its size was rather small compared with what is commonly obtained for speech (cf.…”