A unitary model of auditory frequency change perception

Abstract

Changes in the frequency content of sounds over time are arguably the most basic form of information about the behavior of sound-emitting objects. In perceptual studies, such changes have mostly been investigated separately, as aspects of either pitch or timbre. Here, we propose a unitary account of “up” and “down” subjective judgments of frequency change, based on a model combining auditory correlates of acoustic cues in a sound-specific and listener-specific manner. To do so, we introduce a generalized version of so-called Shepard tones, allowing symmetric manipulations of spectral information on a fine scale, usually associated to pitch (spectral fine structure, SFS), and on a coarse scale, usually associated timbre (spectral envelope, SE). In a series of behavioral experiments, listeners reported “up” or “down” shifts across pairs of generalized Shepard tones that differed in SFS, in SE, or in both. We observed the classic properties of Shepard tones for either SFS or SE shifts: subjective judgements followed the smallest log-frequency change direction, with cases of ambiguity and circularity. Interestingly, when both SFS and SE changes were applied concurrently (synergistically or antagonistically), we observed a trade-off between cues. Listeners were encouraged to report when they perceived “both” directions of change concurrently, but this rarely happened, suggesting a unitary percept. A computational model could accurately fit the behavioral data by combining different cues reflecting frequency changes after auditory filtering. The model revealed that cue weighting depended on the nature of the sound. When presented with harmonic sounds, listeners put more weight on SFS-related cues, whereas inharmonic sounds led to more weight on SE-related cues. Moreover, these stimulus-based factors were modulated by inter-individual differences, revealing variability across listeners in the detailed recipe for “up” and “down” judgments. We argue that frequency changes are tracked perceptually via the adaptive combination of a diverse set of cues, in a manner that is in fact similar to the derivation of other basic auditory dimensions such as spatial location.

Humans use local spectrotemporal correlations to detect rising and falling pitch

“…More generally, listeners can group nearby frequencies into auditory objects, which strongly influences the perception of rising and falling pitch (13). Likewise, the perception of continuity with rising and falling tones is also consistent with tracking auditory objects (14), and psychophysical studies of frequency change detection have tended to use isolated frequencies or persistent sound spectra in which auditory object tracking is possible (15)(16)(17)(18). Studies also show that pitch change discrimination can occur over seconds, suggestive of object tracking (19).…”

Neurophysiological time course of timbre-induced music-like perception

“…This manipulation induced the illusory perception of shifts in timbral brightness analogous to the shifts in pitch induced by the classic Shepard tone. In a follow-up study (Siedenburg et al, 2023), the same Shepard tone variations associated with pitch (shifting spectral fine structure, SFS) and timbral brightness (shifting spectral envelope, SE) were again compared in conditions where both SFS and SE shifts were applied simultaneously in either a synergistic or antagonistic manner (same or opposite directions). Behavioral results showed that the two direction changes were very rarely perceived concurrently, and that the stronger cue did not correlate with the component shifted (SFS or SE), but instead with the nature of the sound itself.…”