Voice emotion recognition by Mandarin‐speaking pediatric cochlear implant users in Taiwan

Abstract

Objectives: To explore the effects of obligatory lexical tone learning on speech emotion recognition and the cross-culture differences between United States and Taiwan for speech emotion understanding in children with cochlear implant.Methods: This cohort study enrolled 60 cochlear-implanted (cCI) Mandarin-speaking, school-aged children who underwent cochlear implantation before 5 years of age and 53 normal-hearing children (cNH) in Taiwan. The emotion recognition and the sensitivity of fundamental frequency (F0) changes for those school-aged cNH and cCI (6-17 years old) were examined in a tertiary referred center. Results:The mean emotion recognition score of the cNH group was significantly better than the cCI. Female speakers' vocal emotions are more easily to be recognized than male speakers' emotion. There was a significant effect of age at test on voice recognition performance. The average score of cCI with full-spectrum speech was close to the average score of cNH with eight-channel narrowband vocoder speech.The average performance of voice emotion recognition across speakers for cCI could be predicted by their sensitivity to changes in F0.Conclusions: Better pitch discrimination ability comes with better voice emotion recognition for Mandarin-speaking cCI. Besides the F0 cues, cCI are likely to adapt their voice emotion recognition by relying more on secondary cues such as intensity and duration. Although cross-culture differences exist for the acoustic features of voice emotion, Mandarin-speaking cCI and their English-speaking cCI peer expressed a positive effect for age at test on emotion recognition, suggesting the learning effect and brain plasticity. Therefore, further device/processor development to improve presentation of pitch information and more rehabilitative efforts are needed to improve the transmission and perception of voice emotion in Mandarin.Level of evidence: 3.

Predictors of Emotional Prosody Identification by School-Age Children With Cochlear Implants and Their Peers With Normal Hearing

“…While previous studies have shown developmental effects as well as sensitivity to talkers and emotions in children with CIs emotional prosody perception (Volkova et al 2013;Wang et al 2019;Barrett et al 2020;Lin et al 2022), the interactive effects between age, cognitive status, emotion categories, and talker have not been studied. The present study builds on previous findings by investigating these interactions.…”

“…This suggests that concepts of different emotions (termed emotion understanding) are developed by the time children with CIs are school age. However, school-age children with CIs still show significant deficits relative to children with normal hearing in vocal emotion recognition tasks (Luo et al 2007;Chatterjee et al 2015;Gilbers et al 2015;Barrett et al 2020;Lin et al 2022). Strong intersubject variability is also evident in performance by children with CIs in vocal emotion identification accuracy, with some of them achieving excellent performance while others' scores are at chance level (Chatterjee et al 2015;Barrett et al 2020).…”

“…How this involvement of cognitive resources might change with increasing age or experience with the device, however, is a question that can only be answered by examining the interaction between cognitive function and age in predicting emotion identification outcomes. The acoustic features of emotional prosody vary across talkers and emotions (Luo et al 2007; Chatterjee et al 2015; Barrett et al 2020; Lin et al 2022). These variations imply that some talkers are better at conveying emotions vocally than others.…”

Vocal and musical emotion perception, voice cue discrimination, and quality of life in cochlear implant users with and without acoustic hearing

“…However, due to the properties of electrode-nerve interface, the transmitted information is reduced in spectrotemporal details (for an overview see Başkent et al, 2016). In particular, the perception of vocal and musical emotion is consistently reported to be challenging for CI users (Ambert-Dahan et al, 2015; Chatterjee et al, 2015, 2023; Lin et al, 2022; Luo et al, 2007; Paquette et al, 2018; Volkova et al, 2013). The ability to perceive vocal emotions in CI users has been shown to be linked to voice cue discrimination thresholds for emotion-related acoustic features such as the voice cues of voice pitch (related to voice fundamental frequency, F0) (Barrett et al, 2020; Deroche et al, 2016; Lin et al, 2022) and vocal tract length (VTL; related to formants, Chuenwattanapranithi et al, 2008; Nussbaum et al, 2022; von Eiff et al, 2022) as well as to listeners’ ratings of quality of life (QoL) (Luo et al, 2018).…”

The effect of lexical tone experience on English intonation perception in Mandarin-speaking cochlear-implanted children

“…Perception of prosody of speech is very important in speech understanding. Prosody has linguistic function (e.g., in lexical tones or intonation), expressive function (in semantic, attitudinal, psychological, and social domains), [18,23,24] and plays an important role in expediting spoken language development. [25][26][27] Mandarin and English are the 2 widely-spoken languages in the present time.…”