Musical Emotion Categorization with Vocoders of Varying Temporal and Spectral Content

Abstract

While previous research investigating music emotion perception of cochlear implant (CI) users observed that temporal cues informing tempo largely convey emotional arousal (relaxing/stimulating), it remains unclear how other properties of the temporal content may contribute to the transmission of arousal features. Moreover, while detailed spectral information related to pitch and harmony in music — often not well perceived by CI users— reportedly conveys emotional valence (positive, negative), it remains unclear how the quality of spectral content contributes to valence perception. Therefore, the current study used vocoders to vary temporal and spectral content of music and tested music emotion categorization (joy, fear, serenity, sadness) in 23 normal-hearing participants. Vocoders were varied with two carriers (sinewave or noise; primarily modulating temporal information), and two filter orders (low or high; primarily modulating spectral information). Results indicated that emotion categorization was above-chance in vocoded excerpts but poorer than in a non-vocoded control condition. Among vocoded conditions, better temporal content (sinewave carriers) improved emotion categorization with a large effect while better spectral content (high filter order) improved it with a small effect. Arousal features were comparably transmitted in non-vocoded and vocoded conditions, indicating that lower temporal content successfully conveyed emotional arousal. Valence feature transmission steeply declined in vocoded conditions, revealing that valence perception was difficult for both lower and higher spectral content. The reliance on arousal information for emotion categorization of vocoded music suggests that efforts to refine temporal cues in the CI user signal may immediately benefit their music emotion perception.

Vocal and musical emotion perception, voice cue discrimination, and quality of life in cochlear implant users with and without acoustic hearing

“…With respect to categories, joy was categorized with the most sensitivity and sadness with the least. This is in line with the overall sensitivity reported with NH participants with the vocoded version of the current musical emotion stimuli (Harding et al, 2023). This finding generally aligns with visual inspection of accuracy scores in Ambert-Dahan et al (2015) (d' and mean scores not reported), where CI users categorized musical emotion in classical excerpts with four emotions: happiness, fear, sadness and peacefulness.…”

“…The findings replicated the results with NH participants tested with vocoders in a previous study (Harding et al, 2023). The d’ range of current adult CI users was similar to the d’ range of vocoded excerpts reported by Harding et al (2023) and below that of the d’ scores with the fully presented, non-vocoded acoustic signal.…”

“…When NH participants were tested with the full acoustic signal in Harding et al (2023), a main effect of emotion showed that for both full acoustic stimuli and vocoded stimuli, joy and fear had higher sensitivity than serenity and sadness. Thus we would expect a similar pattern of sensitivity for CI users.…”

“…Thus we would expect a similar pattern of sensitivity for CI users. Inspection of arousal and valence dimensions in Harding et al (2023) indicated that arousal and valence were both accurately perceived, arousal being perceived slightly better than valence. In contrast, when tested with vocoded excerpts (manipulating the temporal and spectral content available in musical excerpts to acoustically simulate spectrotemporal degradations), perception of arousal decreased slightly while perception of valence was at floor.…”

“…For vocal emotion categorization, we used the EmoHI test, previously used with CI children (Nagels et al, 2020), with three emotion categories of happy, angry, and sad (Experiment 1). For musical emotion categorization, we used a paradigm previously tested on NH participants with CI simulated classical music excerpts, with four emotion categories of joy, fear, serenity and sadness (Filipic et al, 2010; Harding et al, 2023; Lévêque et al, 2018) (Experiment 2). The influence of acoustic hearing was explored by comparing outcomes between unilaterally implanted CI users who reported to have acoustic hearing (bimodal group) or no acoustic hearing (unilateral group) in addition to electric hearing with the CI (Experiment 3a).…”