Auditory and visual scene analysis: an overview

Abstract

We perceive the world as stable and composed of discrete objects even though auditory and visual inputs are often ambiguous owing to spatial and temporal occluders and changes in the conditions of observation. This raises important questions regarding where and how ‘scene analysis’ is performed in the brain. Recent advances from both auditory and visual research suggest that the brain does not simply process the incoming scene properties. Rather, top-down processes such as attention, expectations and prior knowledge facilitate scene perception. Thus, scene analysis is linked not only with the extraction of stimulus features and formation and selection of perceptual objects, but also with selective attention, perceptual binding and awareness. This special issue covers novel advances in scene-analysis research obtained using a combination of psychophysics, computational modelling, neuroimaging and neurophysiology, and presents new empirical and theoretical approaches. For integrative understanding of scene analysis beyond and across sensory modalities, we provide a collection of 15 articles that enable comparison and integration of recent findings in auditory and visual scene analysis. This article is part of the themed issue ‘Auditory and visual scene analysis’.

On Gaze Deployment to Audio-Visual Cues of Social Interactions

“…Burg [65] provided evidence that audio-visual synchrony guides attention in an exogenous manner in adults. However, it remains unclear how multimodal scenes are represented in the brain [66] and there is no comprehensive framework to explain our abilities in multimodal attention.…”

Architecture as Music: A personal journey through time and space

“…How do the myriad of sights and sounds processed by the brain succeed in conjuring up an impression of a piece of music or work of architecture? It is a subject of considerable current concern for neuroscientists who, succinctly, have coined the term Auditory and Visual Scene Analysis to embrace the extraordinarily complex process by which we see and hear (Kondo et al 2017). The unravelling of the mysteries of the human brain provides one of the great challenges in science today.…”

OntoScene, A Logic-Based Scene Interpreter: Implementation and Application in the Rock Art Domain

“…Human perception of complex visual scenes has been studied for a long time in psychology and neuroscience (Kondo et al . 2017): according to the seminal work on "high-level scene perception" (Henderson and Hollingworth 1999), besides low-level or early vision, concerned with extraction of physical properties such as depth, color, and texture from an image (Marr 1982), and intermediate-level vision, concerned with extraction of shape and spatial relations that can be determined without regard to meaning (Ullman 1996), a further level of vision is required to perceive and understand a scene:…”