Cross-Modal Learning in the Auditory System

Bruns, Patrick; Röder, Brigitte

doi:10.1007/978-3-030-10461-0_11

Cited by 6 publications

(3 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In biological systems the hippocampus is thought to be an associative memory "convergence zone," binding together the multimodal elements of an experienced event into a single engram [5]. Our brains are able to remember, integrate and represent information from multiple sensory modalities [6,7,8,9,10,11]. The crossmodal integration is structured such that items can be represented both as a whole as well as a set of crossmodal details [10].…”

Section: Introductionmentioning

confidence: 99%

Various Ways to Recall Cross-modality Perceptual Information in Cognitive Machine

-¹

2023

IJFMR

View full text Add to dashboard Cite

As human, we actively stimulate our memory via variety of recalls for information across different modalities in our brains. The variety of recalls enriches our crossmodal recall mechanism and equip our cognitive system to retrieve information for different needs. Inspired by our cognitive capabilities, this paper formulates, implement and examine different ways in performing crossmodal associative recall of perceptual information in machine cognition. Simulations with different cases and scenarios are conducted and examined on the recalls. Results show that different recalls possess their own features and properties which are useful to equip cognitive machine with variety of cognitive recall capabilities for different purposes.

show abstract

Section: Introductionmentioning

confidence: 99%

Various Ways to Recall Cross-modality Perceptual Information in Cognitive Machine

-¹

2023

IJFMR

View full text Add to dashboard Cite

show abstract

“…Sound is thought to be an effective cue for modulating visual processing in various situations (Shams & Kim, 2010;Spence, 2011). Nevertheless, most studies on audiovisual spatial interactions, such as the spatial ventriloquism effect, have reported that the perceived location of a sound is usually biased toward a spatially disparate visual stimulus (Bruns & Röder, 2019;Chen & Vroomen, 2013). However, it should be noted that audiovisual interactions mostly follow the unity principle, according to which multisensory stimuli that are somewhat discordant in spatial or temporal aspects can be synthesized into a unitary perception (Spence, 2013;Wallace et al, 2004).…”

Section: Introductionmentioning

confidence: 99%

Spatially incongruent sounds affect visual localization in virtual environments

Liu

Rau

2020

Atten Percept Psychophys

View full text Add to dashboard Cite

Distance underestimations along the depth plane are widely found in virtual environments. However, past findings have shown that changes in the visual aspects of virtual reality settings do not lead to more accurate depth estimates. Therefore, we examined if nonvisual stimuli, namely, sounds, could serve as cues that affect observers' depth perception. Accordingly, we conducted two distance discrimination tasks to examine whether observers' depth localization is affected by a spatially incongruent sound. In Experiment 1, a spatially incongruent sound made a visual target appear farther away than a visual target presented with no sound only when a far-distance range (i.e., longer than 12 m) was introduced. Experiment 2 further indicated that the sound shifted visual localization only when audiovisual spatial disparity did not exceed 4°. Taken together, our findings suggest that the depth localization of a visual object in virtual reality can be altered by a spatially incongruent sound, and provide a potential approach that we can adopt a spatially incongruent sound as a cue to reduce the depth compression in VR.

show abstract

“…The brain can represent, integrate, and remember information from more than one sensory modality (Ghazanfar and Schroeder, 2006; Driver and Noesselt, 2008; Bruns and Röder, 2019; Leon et al, 2019; Taesler et al, 2019). This cross-modal integration is structured such that items can be represented both as a whole as well as a set of cross-modal details.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Memory Components and Their Long-Term Dynamics Identified in Cortical Layers II/III but Not Layer V

Liu

Wang

Xie

et al. 2019

Front. Integr. Neurosci.

View full text Add to dashboard Cite

Activity patterns of cerebral cortical regions represent the current environment in which animals receive multi-modal inputs. These patterns are also shaped by the history of activity that reflects learned information on past multimodal exposures. We studied the long-term dynamics of cortical activity patterns during the formation of multimodal memories by analyzing in vivo high-resolution 2-photon mouse brain imaging data of Immediate Early Gene (IEG) expression, resolved by cortical layers. Strikingly, in superficial layers II/III, the patterns showed similar dynamics across structurally and functionally distinct cortical areas and the consistency of dynamic patterns lasted for one to several days. By contrast, in deep layer V, the activity dynamics varied across different areas, and the current activities were sensitive to the previous activities at different time points, depending on the cortical locations, indicating that the information stored in the cortex at different time points was distributed across different cortical areas. These results suggest different roles of superficial and deep layer neurons in the long-term multimodal representation of the environment.

show abstract

Cross-Modal Learning in the Auditory System

Cited by 6 publications

References 82 publications

Various Ways to Recall Cross-modality Perceptual Information in Cognitive Machine

Various Ways to Recall Cross-modality Perceptual Information in Cognitive Machine

Spatially incongruent sounds affect visual localization in virtual environments

Multimodal Memory Components and Their Long-Term Dynamics Identified in Cortical Layers II/III but Not Layer V

Contact Info

Product

Resources

About