Resolving the time course of visual and auditory object categorization

Abstract

Humans can effortlessly categorize objects, both when they are conveyed through visual images and spoken words. To resolve the neural correlates of object categorization, studies have so far primarily focused on the visual modality. It is therefore still unclear how the brain extracts categorical information from auditory signals. In the current study we used EEG (N=47) and time-resolved multivariate pattern analysis to investigate (1) the time course with which object category information emerges in the auditory modality and (2) how the representational transition from individual object identification to category representation compares between the auditory modality and the visual modality. Our results show that (1) that auditory object category representations can be reliably extracted from EEG signals and (2) a similar representational transition occurs in the visual and auditory modalities, where an initial representation at the individual-object level is followed by a subsequent representation of the objects‘ category membership. Altogether, our results suggest an analogous hierarchy of information processing across sensory channels. However, we did not find evidence for a shared supra-modal code, suggesting that the contents of the different sensory hierarchies are ultimately modality-unique.

Influence of social and semantic contexts in processing speech in noise

“…Going further, the influence of context in perception of a stimulus is not only known in the auditory modality but also in the visual modality (Brandman & Peelen, 2017;Oliva & Torralba, 2007), where recent work showed that a social context can have a significant influence on the perception of an individual visual stimulus and its neural processing (Abassi et al, 2020;Bellot et al, 2021). Although the hierarchy of information processing shows analogies in the visual and auditory modalities (Iamshchinina et al, 2022), a significant gap exists in the literature on audition regarding the understanding of the impact of a social context on perceiving verbal sentences, especially in conditions where the auditory environment is challenging. The social context encompasses not only semantic cues but also involves intricate mechanisms related to the interplay between two or more speakers, elements that are generally overlooked in the examination of higher-order top-down cues, and that is a second gap we aimed to fill in the current study.…”