Speech recognition with amplitude and frequency modulations

Abstract

Amplitude modulation (AM) and frequency modulation (FM) are commonly used in communication, but their relative contributions to speech recognition have not been fully explored. To bridge this gap, we derived slowly varying AM and FM from speech sounds and conducted listening tests using stimuli with different modulations in normal-hearing and cochlear-implant subjects. We found that although AM from a limited number of spectral bands may be sufficient for speech recognition in quiet, FM significantly enhances speech recognition in noise, as well as speaker and tone recognition. Additional speech reception threshold measures revealed that FM is particularly critical for speech recognition with a competing voice and is independent of spectral resolution and similarity. These results suggest that AM and FM provide independent yet complementary contributions to support robust speech recognition under realistic listening situations. Encoding FM may improve auditory scene analysis, cochlear-implant, and audiocoding performance.auditory analysis ͉ cochlear implant ͉ neural code ͉ phase ͉ scene analysis A coustic cues in speech sounds allow a listener to derive not only the meaning of an utterance but also the speaker's identity and emotion. Most traditional research has taken a reductionist's approach in investigation of the minimal cues for speech recognition (1). Previous studies using either naturally produced whispered speech (2) or artificially synthesized speech (3, 4) have isolated and identified several important acoustic cues for speech recognition. For example, computers relying on primarily spectral cues and human cochlear-implant listeners relying on primarily temporal cues can achieve a high level of speech recognition in quiet (5-7). As a result, spectral and temporal acoustic cues have been interpreted as built-in redundancy mechanisms in speech recognition (8). However, this redundancy interpretation is challenged by the extremely poor performance of both computers and human cochlear implant users in realistic listening situations where noise is typically present (7, 9).The goal of this study was to delineate the relative contributions of spectral and temporal cues to speech recognition in realistic listening situations. We chose three speech perception tasks that are known to be notoriously difficult for computers and human cochlear-implant users, including speech recognition with a competing voice, speaker recognition, and Mandarin tone recognition. We approached the issue by extracting slowly varying amplitude modulation (AM) and frequency modulation (FM) from a number of frequency bands in speech sounds and testing their relative contributions to speech recognition in acoustic and electric hearing. The AM-only speech has been used in previous studies (3, 10) and is considered to be an acoustic simulation of the cochlear implant (5). Different from previous studies using relatively ''fast'' FM to track formant changes in speech production (4, 11) or fine structure in speech acoustics (12, 13), the ''...

Cochlear implants: A remarkable past and a brilliant future

“…During the past several years, increasing attention has been devoted to representing "fine structure" or "fine frequency" information with CIs (e.g., Smith et al, 2002;Nie et al, 2005;Wilson et al, 2005;Zeng et al, 2005;Hochmair et al, 2006;Arnoldner et al, 2007;Berenstein et al, 2008;Brendel et al, 2008;Buechner et al, 2008;Litvak et al, 2008;Bonham and Litvak, this issue). These recent efforts are reviewed and discussed in Wilson and Dorman (2008b, c).…”

Envelope Coding in Auditory Nerve Fibers Following Noise-Induced Hearing Loss

“…Recent perceptual studies focused on temporal coding have suggested that envelope cues are most salient for understanding speech in quiet (Shannon et al 1995) and fine-structure cues are most important for speech in noise (Qin and Oxenham 2003;Zeng et al 2005). Listeners with SNHL appear to have reduced ability to use fine-structure cues for both speech and non-speech stimuli (Buss et al 2004;Lorenzi et al 2006;Hopkins and Moore 2007), and this deficit does not depend on reduced frequency selectivity (Lorenzi et al 2009;Strelcyk and Dau 2009).…”

Use of Amplitude Modulation Cues Recovered from Frequency Modulation for Cochlear Implant Users When Original Speech Cues Are Severely Degraded

“…From the signal processing stand point, it is generally accepted that two main factors contribute to reduced speech identification in noise for CI users: (1) limited spectral resolution including the negative effect of channel interaction (e.g., Fu et al 1998;Friesen et al 2001;Fu and Nogaki 2005;Won et al 2007; Verschuur 2009), and (2) inability to deliver FM information through current CI speech processors (e.g., Smith et al 2002;Qin and Oxenham 2003;Stickney et al 2005;Zeng et al 2005;Gnansia et al 2010). The latter point should be stated more precisely because speech FM information can be transmitted in the form of recovered AM cues through the CI speech processor and CI users are able to exploit the recovered AM cues for speech identification (Won et al 2012).…”

“…This imposes a rapid deterioration in speech identification performance with increasing levels of background noise (Fu et al 1998;Friesen et al 2001;Fu and Nogaki 2005). An alternative method of delivering speech information is the use of frequency modulation (FM), often referred to as "acoustic temporal fine structure" (Qin and Oxenham 2003;Zeng et al 2005;Drennan et al 2008). FM is attractive because it is less susceptible to background noise (Torick and Keller 1985).…”

“…They tested the hypotheses that: (1) when broadband FM signals are presented, the CI speech processor generates AM cues recovered from broadband FM as a result of the audio-filtering process; and (2) CI users can use the recovered AM cues for robust speech identification. To test these hypotheses, consonant identification was measured for CI users using speech stimuli presented in quiet and processed with FM vocoders (e.g., Zeng et al 2005;Gilbert and Lorenzi 2006;Sheft et al 2008). The FM vocoders preserved the FM speech cues and discarded the AM cues in each frequency analysis band.…”