Separable neural representations of sound sources: Speaker identity and musical timbre

Abstract

Human listeners can quickly and easily recognize different sound sources (objects and events) in their environment. Understanding how this impressive ability is accomplished can improve signal processing and machine intelligence applications along with assistive listening technologies. However, it is not clear how the brain represents the many sounds that humans can recognize (such as speech and music) at the level of individual sources, categories and acoustic features. To examine the cortical organization of these representations, we used patterns of fMRI responses to decode 1) four individual speakers and instruments from one another (separately, within each category), 2) the superordinate category labels associated with each stimulus (speech or instrument), and 3) a set of simple synthesized sounds that could be differentiated entirely on their acoustic features. Data were collected using an interleaved silent steady state sequence to increase the temporal signal-to-noise ratio, and mitigate issues with auditory stimulus presentation in fMRI. Largely separable clusters of voxels in the temporal lobes supported the decoding of individual speakers and instruments from other stimuli in the same category. Decoding the superordinate category of each sound was more accurate and involved a larger portion of the temporal lobes. However, these clusters all overlapped with areas that could decode simple, acoustically separable stimuli. Thus, individual sound sources from different sound categories are represented in separate regions of the temporal lobes that are situated within regions implicated in more general acoustic processes. These results bridge an important gap in our understanding of cortical representations of sounds and their acoustics.

The Rapid Emergence of Auditory Object Representations in Cortex Reflect Central Acoustic Attributes

“…While these acoustic qualities have previously been shown to influence the strength of neural responses on different time scales (Caclin et al, 2006;Giard et al, 1995;Rosburg, 2003;Näätänen & Picton, 1987;Roberts et al, 2000;Alluri et al, 2012;Giordano et al, 2012;Allen et al, 2017;Lewis et al, 2012;Warren et al, 2005;Lewis et al, 2009;Patterson et al, 2002;Menon et al, 2002;Langers et al, 2007;Norman-Haignere et al, 2015;Schönwiesner & Zatorre, 2009;Hullett et al, 2016), our results go further by beginning to link these features to representations of auditory objects and events early in perception. Additionally, we show that these features, which have typically been examined in fMRI (Alluri et al, 2012;Giordano et al, 2012;Allen et al, 2017;Lewis et al, 2012;Warren et al, 2005;Lewis et al, 2009;Patterson et al, 2002;Menon et al, 2002;Langers et al, 2007;Ogg et al, 2019;Norman-Haignere et al, 2015;Schönwiesner & Zatorre, 2009), vary in their influence over time, with more primary acoustic processes in cortex associated with decoding accuracy early in the response such as spectral envelope and modulation power spectra (Norman-Haignere et al, 2015, Chi et al, 2005, Theunissen & Elie, 2014. Later in the response, however, neural representations were more related to spectral and temporal centroid features (with spectral variability and aperiodicity correlating strongly throughout).…”

“…And while musical training has sometimes been found to influence auditory cortical responses (Bidelman, Weiss, Moreno, & Alain, 2014), these effects might arise more robustly during demanding behavioral tasks (see also Alho et al, 2016;Bidelman & Walker, 2017). Indeed, other findings based on passive tasks with diverse, readily distinguishable stimuli have not found an influence of musical training on neural representations (Ogg et al, 2019).…”

“…Additionally, work on classic auditory evoked responses (using simple synthesized stimuli) such as the mismatch negativity (Caclin et al, 2006;Giard et al, 1995;Rosburg, 2003), N1 (Näätänen & Picton, 1987) and M100 response (Roberts, Ferrari, Stufflebeam, & Poeppel, 2000) indicate that the auditory system is sensitive to specific acoustic qualities (e.g., spectral and temporal envelopes, noisiness, spectral variability). The manner by which acoustic features might influence early, time varying neural representations of natural sounds remains largely unexplored despite extensive examination via other methods such as fMRI (Spectral Centroid: Alluri et al, 2012;Giordano, McAdams, Zatorre, Kriegeskorte, & Belin, 2012;Allen, Burton, Olman, & Oxenham, 2017;Spectral Flatness: Alluri et al, 2012, Lewis, Talkington, Tallaksen, & Frum, 2012Spectral Envelope: Warren, Jennings, & Griffiths, 2005;Ogg, Moraczewski, Kuchinsky, & Slevc, 2019;Noisiness: Giordano et al, 2012;Lewis et al, 2009;Fundamental Frequency: Giordano et al, 2012, Allen et al, 2017Patterson, Uppenkamp, Johnsrude, & Griffiths, 2002;Attack Time: Menon et al, 2002;Loudness: Giordano et al, 2012;Langers, van Dijk, Schoenmaker, & Backes, 2007;Spectrotemporal Modulation: Norman-Haignere, Kanwisher, & McDermott, 2015;Schönwiesner & Zatorre, 2009) and ECoG (Hullett, Hamilton, Mesgarani, Schreiner, & Chang, 2016).…”

EEG-based classification of natural sounds reveals specialized responses to speech and music

“…This supports our expectation that the responses 329 are temporally individualized for each sound. Additionally, it suggests that EEG does not pick 330 up the subtle spatial differences in cortical responsiveness to speech and music sounds, as has 331 been found in prior fMRI literature (Giordano et al, 2014;Norman-Haignere et al, 2015, 2019332 14 Ogg et al, 2019), which might be observed with many more speech and music samples in a 333 future experiment. 334…”

Acoustic Correlates of Auditory Object and Event Perception: Speakers, Musical Timbres, and Environmental Sounds

“…To address the issue of time demands, we relied on short duration sounds (250-ms, including the sounds’ onsets). While this may appear to be a quite restricted duration, findings from gating paradigms demonstrate that durations over 200-ms support asymptotic performance for many sound identification tasks (Gray, 1942; Robinson and Patterson, 1995a,b; Suied et al, 2014; Ogg et al, 2017), and this sort of controlled, limited duration has commonly been used in many neuroimaging studies (e.g., Formisano et al, 2008; Leaver and Rauschecker, 2010; Ogg et al, 2019). We focused on sound onsets because they have been shown to carry a large amount of information important for sound perception (Iverson and Krumhansl, 1993; Hjortkjær and McAdams, 2016) and identification (Saldanha and Corso, 1964; Wedin and Goude, 1972; Lemaitre and Heller, 2012; Ogg et al, 2019) and this choice naturally focuses our investigation on the critical timescales when real-world identification is taking place (i.e., after a sound has been initiated in the environment).…”