Aesthetic and physiological effects of naturalistic multimodal music listening

Abstract

Compared to audio only (AO) conditions, audiovisual (AV) information can enhance the aesthetic experience of a music performance. However, such beneficial multimodal effects have yet to be studied in naturalistic music performance settings. Further, peripheral physiological correlates of aesthetic experiences are not well-understood. Here, participants were invited to a concert hall for piano performances of Bach, Messiaen, and Beethoven, which were presented in two conditions: AV and AO. They rated their aesthetic experience (AE) after each piece (Experiment 1 and 2), while peripheral signals (cardio-respiratory measures, skin conductance, and facial muscle activity) were continuously measured (Experiment 2). AE was significantly higher in the AV condition in both experiments. Physiological arousal indices - skin conductance and LF/HF ratio, which represent activation of the sympathetic nervous system - were higher in the AO condition, suggesting increased arousal, perhaps because sound onsets in the AO condition were less predictable. However, breathing was faster and facial muscle activity was higher in the AV condition, suggesting that observing a performer's movements likely enhances motor mimicry in these more voluntary peripheral measures. Further, zygomaticus ('smiling') muscle activity was a significant predictor of AE. Thus, we suggest physiological measures are related to AE, but at different levels: the more involuntary measures (i.e., skin conductance and heart rhythms) may reflect more sensory aspects, while the more voluntary measures (i.e., muscular control of breathing and facial responses) may reflect the liking aspect of an AE. In summary, we replicate and extend previous findings that AV information enhances AE in a more naturalistic music performance setting. We further show that a combination of self-report and peripheral measures benefit a meaningful assessment of AE in naturalistic music performance settings.

Audio-visual concert performances synchronize an audience’s heart rates

“…Participants, stimuli, and procedures are identical to Czepiel et al 9 (see General Methods and Experiment 2). Key details of the procedure are outlined below.…”

“…The visual component has indeed been shown to have a consistent effect on music performance evaluation 8 . Although we have recently replicated this effect outside laboratory settings, showing that audio-visual (AV) performances led to stronger aesthetic appreciation than audio-only (AO) performances in a concert context 9 , these evaluations were recorded at the end of 7-12 minute long musical pieces. An important next step is to explore how the engagement of listeners varies over time, in both AO and AV conditions.…”

“…AO versions were recorded prior to the concert and were presented as recordings through high-quality loudspeakers, while AV versions were performed live by the pianist during the concert. A trained sound engineer checked that the sound level was equal across all stimuli; additional analyses of acoustic features showed these performances were comparable across concerts 9 .…”

“…For the second hypothesis, time-averaged synchrony measures were compared between AV and AO. Self-reported engagement (translated as 'absorption' in a previous study 9 , but used synonymously with engagement, see also 20 ) was also compared between these conditions. Linear mixed models (LMMs) with a fixed effect of modality were constructed for each of the synchrony measures as the outcome variables.…”

“MusicLab Copenhagen”: The Gains and Challenges of Radically Interdisciplinary Concert Research

“…The pioneering “research concerts” of recent decades represent prime examples of interdisciplinary music research due to their multifaceted intersections with psychology, sociology, mathematics, computing, acoustics, medicine, and biology (Klein & Parncutt, 2010). Often they have used a mix of methods (Seibert et al, 2020) to investigate diverse topics such as emotion and aesthetic experience (Coutinho & Scherer, 2017; Czepiel et al, 2023; McAdams et al, 2004; Merrill et al, 2023; Stevens et al, 2014; Tschacher et al, 2023; Thompson, 2006), expectation (Egermann et al, 2013), psychophysiology (Bernardi et al, 2017; Czepiel et al, 2021; Egermann et al, 2013; Sato et al, 2017), movement (Swarbrick et al, 2019), synchrony (Czepiel et al, 2021, 2023; Seibert et al, 2019; Tschacher et al, 2023), joint action (Chang et al, 2017, 2019), and social connection (Swarbrick et al, 2021) during the live concert experience. Whereas many such concert studies have been initiated and spearheaded by psychologically oriented researchers, the MusicLab Copenhagen concert, which we will discuss here, was initiated by humanities scholar and philosopher Simon Høffding (SH) with the explicit goal of obtaining a deeper understanding of joint, embodied concert absorption among both audience members and musicians.…”