Emergence of neural encoding of auditory objects while listening to competing speakers

Abstract

A visual scene is perceived in terms of visual objects. Similar ideas have been proposed for the analogous case of auditory scene analysis, although their hypothesized neural underpinnings have not yet been established. Here, we address this question by recording from subjects selectively listening to one of two competing speakers, either of different or the same sex, using magnetoencephalography. Individual neural representations are seen for the speech of the two speakers, with each being selectively phase locked to the rhythm of the corresponding speech stream and from which can be exclusively reconstructed the temporal envelope of that speech stream. The neural representation of the attended speech dominates responses (with latency near 100 ms) in posterior auditory cortex. Furthermore, when the intensity of the attended and background speakers is separately varied over an 8-dB range, the neural representation of the attended speech adapts only to the intensity of that speaker but not to the intensity of the background speaker, suggesting an object-level intensity gain control. In summary, these results indicate that concurrent auditory objects, even if spectrotemporally overlapping and not resolvable at the auditory periphery, are neurally encoded individually in auditory cortex and emerge as fundamental representational units for topdown attentional modulation and bottom-up neural adaptation.spectrotemporal response function | reverse correlation | phase locking | selective attention I n a complex auditory scene, humans and other animal species can perceptually detect and recognize individual auditory objects (i.e., the sound arising from a single source), even if strongly overlapping acoustically with sounds from other sources. To accomplish this remarkably difficult task, it has been hypothesized that the auditory system first decomposes the complex auditory scene into separate acoustic features and then binds the features, as appropriate, into auditory objects (1-4). The neural representations of auditory objects, each the collective representation of all the features belonging to the same auditory object, have been hypothesized to emerge in auditory cortex to become fundamental units for high-level cognitive processing (5-7). The process of parsing an auditory scene into auditory objects is computationally complex and cannot as yet be emulated by computer algorithms (8), but it occurs reliably, and often effortlessly, in the human auditory system. For example, in the classic "cocktail party problem," where multiple speakers are talking at the same time (9), human listeners can selectively attend to a chosen target speaker, even if the competing speakers are acoustically more salient (e.g., louder) or perceptually very similar (such as of the same sex) (10).To demonstrate an object-based neural representation that could subserve the robust perception of an auditory object, several key pieces of evidence are needed. The first is to demonstrate neural activity that exclusively represents a single auditory...

Effects of Spectral Degradation on Attentional Modulation of Cortical Auditory Responses to Continuous Speech

“…Consistent with previous electrophysiological findings ( Ding and Simon 2012a;Horton et al 2013;Kong et al 2014;O'Sullivan et al 2014), selective attention differentially modulates the neural responses to attended and unattended speech streams for highest resolution spectral conditions (i.e., unproc and 64-ch vocoder) in a challenging 0-dB TMR condition. We argue that high spectral resolution can support sound segregation and, as a result, selective attention can operate on the individually perceived auditory objects, enhancing the neural representation of the attended speech and suppressing the neural representation of unattended speech (Horton et al 2013;Kong et al 2014).…”

“…It could be that the listeners habituated to the stimulation during the course of the test session. However, we argue that this explanation is unlikely because (1) we found no difference in patterns of neural responses between the averaged data for the first five repetitions and the second five repetitions of the same speech mixture, (2) we found no significant difference between TMR conditions for the unprocessed speech (also see Ding and Simon 2012a;Kong et al 2014), but this was not the case for the vocoder speech with lower spectral resolution, (3) the stimuli were different between the 3-and 0-dB TMR conditions, and (4) a mandatory Table 3 The amplitude (i.e., the absolute maximum correlation value) of the peaks in the cross-correlation functions for the attended and unattended speech in different listening conditions break of at least 10 min was given to each subject after each block of testing.…”

“…Previous studies (Ding and Simon 2012a;Kong et al 2014) found that, in a complex auditory scenario consisting of two competing speakers, cortical responses are largely invariant to the intensity ratio between the speakers when it varies in a range where the listeners can successfully attend to the target speaker. However, in the present study, we observed a TMR effect for the neural responses across vocoder conditions, which may imply that the listeners actively use sound intensity as a cue in this experiment.…”

“…Ding and Simon (2012a) varied the attended and background speakers separately over a wide range of intensity and found that the neural representation of the attended speech did not adapt to the intensity of the background speaker, which strongly suggests an object-level intensity gain control. The attentional effect was observed at both early (G100 ms) (Ding and Simon 2012a;Kong et al 2014) and later (9150 ms) (Horton et al 2013;O'Sullivan et al 2014;Kong et al 2014) processing stages.…”

“…Ding and Simon (2012a) varied the attended and background speakers separately over a wide range of intensity and found that the neural representation of the attended speech did not adapt to the intensity of the background speaker, which strongly suggests an object-level intensity gain control. The attentional effect was observed at both early (G100 ms) (Ding and Simon 2012a;Kong et al 2014) and later (9150 ms) (Horton et al 2013;O'Sullivan et al 2014;Kong et al 2014) processing stages. By cross-correlating the electroencephalography (EEG) responses with the envelopes of individual speakers in a two-speaker speech mixture, Kong et al (2014) showed salient peaks around 100 and 200 ms in the cross-correlation function for the attended speech but a delayed and much attenuated peak at around 200 ms for the unattended speech.…”