Neural processing of natural sounds

Theunissen, Frédéric E.; Elie, Julie E.

doi:10.1038/nrn3731

Cited by 208 publications

(189 citation statements)

References 136 publications

Supporting

Mentioning

173

Contrasting

Unclassified

Order By: Relevance

“…To minimize the role of source priors in our separation experiments, we used random synthetic sources with little structure. However, inference could be aided by prior knowledge of the regularities of natural sound sources (51), such that performance on tasks requiring estimates of source and filter might further improve with more naturalistic sources.…”

Section: Discussionmentioning

confidence: 99%

Statistics of natural reverberation enable perceptual separation of sound and space

Traer

McDermott

2016

Proc. Natl. Acad. Sci. U.S.A.

129

130

View full text Add to dashboard Cite

In everyday listening, sound reaches our ears directly from a source as well as indirectly via reflections known as reverberation. Reverberation profoundly distorts the sound from a source, yet humans can both identify sound sources and distinguish environments from the resulting sound, via mechanisms that remain unclear. The core computational challenge is that the acoustic signatures of the source and environment are combined in a single signal received by the ear. Here we ask whether our recognition of sound sources and spaces reflects an ability to separate their effects and whether any such separation is enabled by statistical regularities of real-world reverberation. To first determine whether such statistical regularities exist, we measured impulse responses (IRs) of 271 spaces sampled from the distribution encountered by humans during daily life. The sampled spaces were diverse, but their IRs were tightly constrained, exhibiting exponential decay at frequency-dependent rates: Mid frequencies reverberated longest whereas higher and lower frequencies decayed more rapidly, presumably due to absorptive properties of materials and air. To test whether humans leverage these regularities, we manipulated IR decay characteristics in simulated reverberant audio. Listeners could discriminate sound sources and environments from these signals, but their abilities degraded when reverberation characteristics deviated from those of real-world environments. Subjectively, atypical IRs were mistaken for sound sources. The results suggest the brain separates sound into contributions from the source and the environment, constrained by a prior on natural reverberation. This separation process may contribute to robust recognition while providing information about spaces around us.natural scene statistics | auditory scene analysis | environmental acoustics | psychophysics | psychoacoustics P erception requires the brain to determine the structure of the world from the energy that impinges upon our sensory receptors. One challenge is that most perceptual problems are illposed: The information we seek about the world is underdetermined given the sensory input. Sometimes this is because noise partially obscures the structure of interest. In other cases, it is because the sensory signal is influenced by multiple causal factors in the world. In vision, the light that enters the eye is a function of the surface pigmentation we typically need to estimate, but also of the illumination level. In touch, estimates of surface texture from vibrations are confounded by the speed with which a surface passes over the skin's receptors. And in hearing, we seek to understand the content of individual sound sources in the world, but the ear often receives a signal that is a mixture of multiple sources. These problems are all examples of scene analysis, in which the brain must infer one or more of the multiple factors that created the signals it receives (1). Inference in such cases is possible only with the aid of prior information about the world...

show abstract

Section: Discussionmentioning

confidence: 99%

Statistics of natural reverberation enable perceptual separation of sound and space

Traer

McDermott

2016

Proc. Natl. Acad. Sci. U.S.A.

129

130

View full text Add to dashboard Cite

show abstract

“…How complex sounds are further transformed and analyzed along the auditory neural pathway and in the cortex remains uncertain. Ethological considerations have led to the hypothesis that brain processing of sounds is optimized for spectrotemporal modulations, which are characteristically present in ecologically relevant sounds (1), such as in animal vocalizations [e.g., zebra finch (2), macaque monkeys (3)] and speech (2,(4)(5)(6)(7). Modulations are regular variations of energy in time, in frequency, or in time and frequency simultaneously.…”

mentioning

confidence: 99%

Reconstructing the spectrotemporal modulations of real-life sounds from fMRI response patterns

Santoro

Moerel

Martino

et al. 2017

Proc. Natl. Acad. Sci. U.S.A.

104

152

View full text Add to dashboard Cite

Ethological views of brain functioning suggest that sound representations and computations in the auditory neural system are optimized finely to process and discriminate behaviorally relevant acoustic features and sounds (e.g., spectrotemporal modulations in the songs of zebra finches). Here, we show that modeling of neural sound representations in terms of frequency-specific spectrotemporal modulations enables accurate and specific reconstruction of real-life sounds from high-resolution functional magnetic resonance imaging (fMRI) response patterns in the human auditory cortex. Region-based analyses indicated that response patterns in separate portions of the auditory cortex are informative of distinctive sets of spectrotemporal modulations. Most relevantly, results revealed that in early auditory regions, and progressively more in surrounding regions, temporal modulations in a range relevant for speech analysis (∼2-4 Hz) were reconstructed more faithfully than other temporal modulations. In early auditory regions, this effect was frequency-dependent and only present for lower frequencies (<∼2 kHz), whereas for higher frequencies, reconstruction accuracy was higher for faster temporal modulations. Further analyses suggested that auditory cortical processing optimized for the fine-grained discrimination of speech and vocal sounds underlies this enhanced reconstruction accuracy. In sum, the present study introduces an approach to embed models of neural sound representations in the analysis of fMRI response patterns. Furthermore, it reveals that, in the human brain, even general purpose and fundamental neural processing mechanisms are shaped by the physical features of real-world stimuli that are most relevant for behavior (i.e., speech, voice).any natural and man-made sources in our environment produce acoustic waveforms (sounds) consisting of complex mixtures of multiple frequencies (Fig. 1A). At the cochlea, these waveforms are decomposed into frequency-specific temporal patterns of neural signals, typically described with auditory spectrograms (Fig. 1B). How complex sounds are further transformed and analyzed along the auditory neural pathway and in the cortex remains uncertain. Ethological considerations have led to the hypothesis that brain processing of sounds is optimized for spectrotemporal modulations, which are characteristically present in ecologically relevant sounds (1), such as in animal vocalizations [e.g., zebra finch (2), macaque monkeys (3)] and speech (2, 4-7). Modulations are regular variations of energy in time, in frequency, or in time and frequency simultaneously. Typically, in natural sounds, modulations dynamically change over time. The contribution of modulations to sound spectrograms can be made explicit using 4D (time-varying; Movies S1-S3, Left) or 3D (time-averaged; Fig. 1C) representations. Such representations highlight, for example, that the energy of human vocal sounds is mostly concentrated at lower frequencies and at lower temporal modulation rates (Fig. 1C, Left), whereas the...

show abstract

“…This choice helps avoid the risk that other higher level attributes of the sounds could dominate the measured effects. Consistent with Theunissen and Elie [38], our hypothesis is that the features with statistically significant effects on listeners' event-detection performance are also more perceptually salient within the set of all possible features: they emerge naturally from the use of uncontrolled, yet biologically relevant, stimuli.…”

Section: Corpora Of Natural Sounds: Motivation and Designmentioning

confidence: 60%

“…At present, we cannot generalize our results about the feature set and their relative importance to scenarios with episodic sounds (those that happen only once), or with very slow patterns, when the tempo perception breaks down. Moreover, replicating these results using classes of natural sounds other than chirps and different tempi would be beneficial [38].…”

Section: Discussionmentioning

confidence: 96%

“…Theunissen and Elie [38] propose the use of natural sounds to probe the auditory system and introduce an analytic framework for the characterization of the auditory stimulus-response using animal vocalizations. Moreover, natural sounds are better accepted than synthetic sounds, especially when the application requires prolonged listening [18], as with most sonification designs for continuous process monitoring [16].…”

Section: Corpora Of Natural Sounds: Motivation and Designmentioning

confidence: 99%

See 1 more Smart Citation

Prioritizing foreground selection of natural chirp sounds by tempo and spectral centroid

Tordini

Bregman

Cooperstock

2016

J Multimodal User Interfaces

View full text Add to dashboard Cite

Salience shapes the involuntary perception of a sound scene into foreground and background. Auditory interfaces, such as those used in continuous process monitoring, rely on the prominence of those sounds that are perceived as foreground. We propose to distinguish between the salience of sound events and that of streams, and introduce a paradigm to study the latter using repetitive patterns of natural chirps. Since streams are the sound objects populating the auditory scene, we suggest the use of global descriptors of perceptual dimensions to predict their salience, and hence, the organization of the objects into foreground and background. However, there are many possible independent features that can be used to describe sounds. Based on the results of two experiments, we suggest a parsimonious interpretation of the rules guiding foreground formation: after loudness, tempo and brightness are the dimensions that have higher priority. Our data show that, under equal-loudness conditions, patterns with fast tempo and lower brightness tend to emerge and that the interaction between tempo and brightness in foreground selection seems to increase with task difficulty. We propose to use the relations we uncovered as the underpinnings for a computational model of foreground selection, and also, as design guidelines for stream-based sonification applications.

show abstract

Neural processing of natural sounds

Cited by 208 publications

References 136 publications

Statistics of natural reverberation enable perceptual separation of sound and space

Statistics of natural reverberation enable perceptual separation of sound and space

Reconstructing the spectrotemporal modulations of real-life sounds from fMRI response patterns

Prioritizing foreground selection of natural chirp sounds by tempo and spectral centroid

Contact Info

Product

Resources

About