Optical Phonetics and Visual Perception of Lexical and Phrasal Stress in English

Abstract

In a study of optical cues to the visual perception of stress, three American English talkers spoke words that differed in lexical stress and sentences that differed in phrasal stress, while video and movements of the face were recorded. The production of stressed and unstressed syllables from these utterances was analyzed along many measures of facial movement, which were generally larger and faster in the stressed condition. In a visual perception experiment, 16 perceivers identified the location of stress in forced-choice judgments of video clips of these utterances (without audio). Phrasal stress was better perceived than lexical stress. The relation of the visual intelligibility of the prosody of these utterances to the optical characteristics of their production was analyzed to determine which cues are associated with successful visual perception. While most optical measures were correlated with perception performance, chin measures, especially Chin Opening Displacement, contributed the most to correct perception independently of the other measures. Thus, our results indicate that the information for visual stress perception is mainly associated with mouth opening movements.

English Listeners Use Suprasegmental Cues to Lexical Stress Early During Spoken-Word Recognition

“…van Heuven & Menert, 1996). One possibility is that this asymmetry is driven by the influence of lexical stress on intelligibility: Stressed syllables are produced with more precise and extensive articulation (e.g., Scarborough, Keating, Mattys, Cho, & Alwan, 2009), are more informative about the identity of a word, and are more reliably recognized than unstressed syllables (e.g., Altman & Carter, 1989;Lieberman, 1963;McAllister, 1991;Mehta & Cutler, 1988). Another possible explanation for this asymmetry can be the placement of pitch accents on primary-stressed syllables.…”

Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions

“…the face, has led to a new discipline which is sometimes referred to as ''optical phonetics'' (Scarborough et al, 2009). An important problem for this field concerns the relationship between auditory and visual information in human communication.…”

“…Approaches that base their analyses on spontaneous speech data have often been criticized for using anecdotal evidence. To counter such problems, there have been a number of laboratory studies into the functional use of facial expressions, either by eliciting such expressions from real speakers in specifically designed linguistic or social contexts (Dohen, 2005;Dohen & Loevenbruck, 2009;Dohen, Loevenbruck, Cathiard, & Schwartz, 2004;Krahmer & Swerts, 2007;Scarborough et al, 2009;, or by using an analysis-by-synthesis procedure whereby the effect of specific variations in the face on observers are tested through controlled manipulations of synthetic talking heads (Cassell, Vihjá lmsson, & Bickmore, 2001;Granström, House, & Lundeberg, 1999;Krahmer, Ruttkay, Swerts, & Wesselink, 2002;Krahmer & Swerts, 2004;Pelachaud, Badler, & Steedman, 1996). Such studies showed that facial expressions are indeed functionally relevant as they are exploited to support a broad range of communicative functions (e.g., to highlight important information, or to reveal attitudinal or emotional correlates of speaker utterances).…”

Emphatic Visual Speech Synthesis

“…This idea has been later enforced [34]- [37]. Specifically, [34] related eyebrow movement to pitch variations, [35] concluded that stress was best perceived by the visual channel, [36] studied particular articulators involved with stress, and [37] investigated different facial regions. Moreover, facial movements have an influence on the perceived message and its related emotion.…”