Cross-modal interactions in the experience of musical performances: Physiological correlates

Abstract

This experiment was conducted to investigate cross-modal interactions in the emotional experience of music listeners. Previous research showed that visual information present in a musical performance is rich in expressive content, and moderates the subjective emotional experience of a participant listening and/or observing musical stimuli [Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80–113.]. The goal of this follow-up experiment was to replicate this cross-modal interaction by investigating the objective, physiological aspect of emotional response to music measuring electrodermal activity [...

Interaction of Sight and Sound in the Perception and Experience of Musical Performance

“…Although previous studies have shown that visual information can influence the emotions induced by a musical performance (e.g., Chapados & Levitin, 2008;Krahé et al, 2013;Timmers, Marolt, Camurri, & Volpe, 2006, Vines et al, 2011, they haven't been able to reliably estimate the effects size of visual performance cues relative to that of auditory performance cues. The present study revealed that -in terms of the emotional impact of musical performances -the contribution of visual kinematic performance cues appears to be comparable to that of auditory performance cues.…”

“…Hargreaves & North, 2010). Although there is some evidence to indicate that visual information might enhance emotional reactivity to musical performances (Chapados & Levitin, 2008), it is not yet known how the effect of visual performance cues relates to that of auditory performance cues with regard to the subjective emotional reaction of the observer. Furthermore, the explicit instructions used by Vuoskoski et al to take both auditory and visual aspects of the performance into account in the evaluations of overall expressivity might have affected which aspects of the material the participants attended to (for details, see Vuoskoski et al, 2014; Experiment 1).…”

“…The former experiment investigated evaluations of a perceived characteristic of the performances (i.e., perceived expressivity), while the current experiment investigates the subjective emotional reactions experienced by participants (i.e., felt emotion). Previous research has suggested that visual information may have a significant impact on an observer's emotional reactions to a musical performance (Chapados & Levitin, 2008;Krahé et al, 2013;Vines et al, 2006), but the effect size of visual kinematic performance cues relative to that of auditory performance cues has yet to be investigated.…”

“…In musical performance, a performer's body movements and gestures can convey a range of meaningful information to audiences and co-performers alike, including emotional expression (Castellano, Mortillaro, Camurri, Volpe, & Scherer, 2008;Dahl & Friberg, 2007;Davidson, 1993Davidson, , 1994 and phrasing (Juchniewicz, 2008;Vines, Krumhansl, Wanderley, & Levitin, 2006), as well as musical ideas and timing (Glowinski et al, 2013;Goebl & Palmer, 2009;Williamon & Davidson, 2002). The salience of visual kinematic information (i.e., visual information about performers' body movements and gestures) for an observer's perception and experience of a musical performance has been widely documented (e.g., Chapados & Levitin, 2008;Davidson, 1993;Tsay, 2013;Vines, Krumhansl, Wanderley, Dalca, & Levitin, 2011;Vines et al, 2006), and a recent meta-analysis by Platz and Kopiez (2012) revealed that, compared to audio-only presentations, audiovisual information has a moderate effect on participants' evaluations of a musical performance.…”

Hearing How Smooth It Looks: Selective Attention and Crossmodal Perception in the Arts

“…31 EDA generally followed similar contours in all three conditions. Correlations between EDA and subjective measures of tension from the original study were strongest for the AV condition.…”