Audio-visual saliency map: Overview, basic models and hardware implementation

Abstract

In this paper we provide an overview of audio visual saliency map models. In the simplest model, the location of auditory source is modeled as a Gaussian and use different methods of combining the auditory and visual information. We then provide experimental results with applications of simple audio-visual integration models for cognitive scene analysis. We validate the simple audio-visual saliency models with a hardware convolutional network architecture and real data recorded from moving audio-visual objects. The latter system was developed under Torch language by extending the attention.lua (code) and attention.ui (GUI) files that implement Culurciello's visual attention model. I. I NTRODUCTIONScientists and engineers have traditionally separated the analysis of a multisensory scene into its constituent sensory domains. In this approach, for example, all auditory events are processed separately and independently of visual and somatosensory streams even though the same multisensory event may give rise to those constituent streams. It was previously necessary to compartmentalize the analysis because of the sheer enonnity of information as well as the limita tions of experimental techniques and computational resources. With recent advances in science and technology, it is now possible to perform integrated analysis of sensory systems including interactions within and across sensory modalities. Such efforts are becoming increasingly common in cellular neurophysiology, imaging and psychophysics studies [1], [2]. A better understanding of interaction, information integration, and complementarity of information across senses may help us build many intelligent algorithms for object detection, object recognition, human activity and gait detection, surveillance, tracking, biometrics etc, with better performance, stability and robustness to noise. For example, fusing auditory (voice) and visual (face) features can help improve the perfonnance of speaker identification and face recognition systems [3], [4].There are several examples of highly successful neuromor phic engineering systems [5], [6] that mimic the function of individual sensory systems. However, the efforts have so far been limited to modeling only individual sensory systems rather than the interaction between them. Our goal in this work is to build computational models of multisensory processing to analyze real world perceptual scenes. We limit our focus to two important sensory systems: the visual and auditory systems. Our work is divided into two parts, one being computational verification and the other being hardware implementation. We investigate the nature of multisensory interaction between the auditory and visual domains. More specifically, we consider the effect of a spatially co-occurring auditory stimulus on the salience of an inconspicuous visual target at the same spatial location among other visual distractors. Temporal concurrency is assumed between visual and auditory events. The motiva tion for this work is that audio-visual integration is hi...

Modelling auditory attention

“…In recent years, the necessity of incorporating auditory salience information in visual attention models is becoming increasingly recognized. This has led to the emergence of models using auditory salience direction to guide visual attention [89–90], along with audio–visual models where the two domains have equal weight in determining attentional orientation [91]. These models show better performance than visual-only salience models in predicting eye gazes in videos.…”

“…R. Soc. B 372: 20160101 [89 -90], along with audio -visual models where the two domains have equal weight in determining attentional orientation [91]. These models show better performance than visual-only salience models in predicting eye gazes in videos.…”

A Biologically Motivated, Proto-Object-Based Audiovisual Saliency Model

“…The study of multi-sensory integration [3,6,17], specifically audiovisual integration [4,18], has been an active area of research in neuroscience, psychology and cognitive science. In the computer science and engineering fields, there is an increased interest in the recent times [19][20][21]. For a detailed review of neuroscience and psychophysics research related to audiovisual interaction, please refer to [3,17].…”

Οπτικοακουστική Μοντελοποίηση Προσοχής Και Πολυ-Αισθητηριακή Κατανόηση Ακουστικής Σκηνής

“…Ïpou x, y e-nai oi suntetagmËnec twn eikonostoiqe-wn thc eikÏnac, to F T A e-nai o sundua-smËnoc qàrthc, to F T antiproswpe'ei to qàrth emfàneiac enÏc memonwmËnou qroniko' optiko' qarakthristiko' kai A e-nai h akoustik † kamp'lh emfàneiac. Aut † h idËa prwtoemfan-sthke me Ëna parÏmoio allà Ïqi tautÏshmo trÏpo sto [175]. Oi suggrafe-c se aut † th douleià peiramat-zontai mÏno me qwrik † akoustik † emfàneia kai o akoustikÏc qàrthc emfàneiac pou paràgoun e-nai mÏno h jËsh thc akoustik †c phg †c sto q∏ro.…”

“…Sto [175] o akoustikÏc qàrthc upolog-zetai ep-shc me entopismÏ jËshc phg †c kai sth sunËqeia sugqwne'etai me ton optikÏ mËsw pollaplasiasmo'. Sto [176] parousiàzetai Ëna trisdiàstato optikoakoustikÏ montËlo prosoq †c to opo-o sunduàzei optikà dedomËna, de-domËna bàjouc kai polukanalikà akoustikà dedomËna mËsw d'o anexàrthtwn statik∏n kai dunamik∏n drÏmwn anàlushc, en∏ sto [177] perigràfetai Ëna optikoakoustikÏ montËlo pou prospaje-na problËyei to anjr∏pino blËmma (gaze) se efarmogËc thlediàskeyhc.…”