Perceiving affect from the voice and the face

Abstract

This experiment examines how emotion is perceived by using facial and vocal cues of a speaker. Three levels of facial affect were presented using a computer-generated face. Three levels of vocal affect were obtained by recording the voice of a male amateur actor who spoke a semantically neutral word in different simulated emotional states. These two independent variables were presented to subjects in all possible permutations--visual cues alone, vocal cues alone, and visual and vocal cues together-which gave a total set of 15 stimuli. The subjects were asked to judge the emotion of the stimuli in a two-alternative forced choice task (either HAPPy or ANGRY). The results indicate that subjects evaluate and integrate information from both modalities to perceive emotion. The influence of one modality was greater to the extent that the other was ambiguous (neutral). The fuzzy logical model of perception (FLMP)fit the judgments significantly better than an additive model, which weakens theories based on an additive combination of modalities, categorical perception, and influence from only a single modality.Research has shown that we use multiple sources of information when we comprehend speech (Massaro, 1987b(Massaro, , 1989Massaro & Cohen, 1990). Visual information from a speaker's face, for example, can strongly influence speech perception, especially when the auditory information is degraded: in one study, recognition of auditory sentences in noisy environments improved from 23% to 65% when the perceivers could also see the speaker's face (Summerfield, 1979). We also use multiple sources of information when we perceive a speaker's emotion. These sources include a variety ofparalinguistic signals, as well as the speech's verbal content. The emotion may be interpreted in different ways, depending on the voice quality, facial expression, and body language ofthe speaker. To study the degree to which paralinguistic sources of information are used, it is important that one first define these sources and then determine how they are evaluated and integrated. In the present study, in order to investigate the perception of a speaker's emotion, two sources of paralinguistic information were varied: facial expressions and vocal cues.Facial expressions are an effective means of communicating emotion. Darwin (1872) argued that facial expressions originate in basic acts of self-preservation common to human beings and other animals, and that these expressions are related to the emotional states that they convey. Research by Meltzoff and Moore (1977) suggests that we are biologically prepared from birth to respond to facial expressions. They produced evidence which showed that

Crossmodal transfer of emotion by music

“…It was shown earlier [14] that as compared to emotionally distinct (i.e. happy, angry, sad) facial stimuli, emotionally ambiguous (i.e.…”

On Being Consistent: The Role of Verbal–Nonverbal Consistency in First Impressions

“…For example, infant judgments of others' emotions depend on the consistency of others' behavior, such as consistent facial and vocal behavior (Kahana-Kalman & Walker-Andrews, 2001;Walker-Andrews, 1997, 2008. Even after people reach adulthood, behavioral coherence remains important: Adult judgments of facial emotion are strongly disrupted by inconsistent emotional information from other channels of communication (Aviezer et al, 2008;de Gelder & Vroomen, 2000;de Gelder, Bocker, Tuomainen, Hensen, & Vroomen, 1999;Ethofer et al, 2006;Massaro & Egan, 1996;Van den Stock, Grezes, & de Gelder, 2008). In fact, adults often believe that others are being deceptive when the words and nonverbal behaviors of those others are inconsistent (Fleming & Rudman, 1993;Heinrich & Borkenau, 1998;Zuckerman, Driver, & Koestner, 1982).…”

“…The importance of coherence extends to adulthood. For example, recent research has demonstrated that emotions that are clearly expressed in one channel (e.g., the face) are difficult to interpret when another channel (e.g., the body) presents inconsistent emotional information (Aviezer et al, 2008;de Gelder et al, 1999;de Gelder & Vroomen, 2000;Ethofer et al, 2006;Massaro & Egan, 1996;Van den Stock et al, 2008).…”

Coherent activity between auditory and visual modalities during the induction of peacefulness

“…In the latter part, we tried to investigate (1) the role of theta oscillations during peaceful mood, (2) the relations of varied modalities with alpha and beta oscillations, respectively, and (3) the increase of gamma oscillations to the audiovisual modality. Besides, although some behavioral studies used dynamic visual and vocal clips of affect expressions and found those more appropriate in real life (Massaro and Egan 1996;Campanella and Belin 2007;Collignon et al 2008), most related psychophysiological and electrophysiological studies have compared static pictures with dynamic sound (Pourtois et al 2002;Koelewijn et al 2010), which have incompatible dimensions. According to that, the present study focused on how inner quiet information is processed in the human brain with both dynamic materials (i.e.…”