A GPU based saliency map for high-fidelity selective rendering

Abstract

The computation of high-fidelity images in real-time remains one of the key challenges for computer graphics. Recent work has shown that by understanding the human visual system, selective rendering may be used to render only those parts to which the human viewer is attending at high quality and the rest of the scene at a much lower quality. This can result in a significant reduction in computational time, without the viewer being aware of the quality difference. Selective rendering is guided by models of the human visual system, typically in the form of a 2D saliency map, which predict where the user will be looking in any scene. Computation of these maps themselves often take many seconds, thus precluding such an approach in any interactive system, where many frames need to be rendered per second. In this paper we present a novel saliency map which exploits the computational performance of modern GPUs. With our approach it is thus possible to calculate this map in milliseconds, allowing it to be part of a real time rendering system. In addition, we also show how depth, habituation and motion can be added to the saliency map to further guide the selective rendering. This ensures that only the most perceptually important parts of any animated sequence need be rendered in high quality. The rest of the animation can be rendered at a significantly lower quality, and thus much lower computational cost, without the user being aware of this difference.

Visual saliency guided textured model simplification

“…For instance, [29] extracts view-dependent ridgevalley feature lines to form object illustration. On the other hand, saliency detection can be generalised to identify important 3D scene parts by processing model space motion and depth information as well as image space colour and luminance information [25]. A further application is realtime tracking of visually attended 3D scene models, where [24] performed this based on both image features (colour and luminance) and 3D model features (depth, model size, and model motion).…”

Design and Application of Real-Time Visual Attention Model for the Exploration of 3D Virtual Environments

“…For example, Yarbus [10] has shown that the way people look at pictures strongly depends on the task they have to achieve. Furthermore, the top-down component is subject to the habituation phenomenon [11], i.e. objects become familiar over time, and we become oblivious to them.…”

“…objects become familiar over time, and we become oblivious to them. Several models have been proposed to simulate the multiple top-down components using task-map [8], habituation [11], memory [12] as well as spatio-temporal contexts [6].…”

“…The bottom-up component of our model computes a pixel-level saliency map using several visual features: intensity [4], antagonistic colors [4], depth [11] and motion [5].…”

Acoustic Rendering and Auditory–Visual Cross‐Modal Perception and Interaction

“…Used in angular sensitivity (James, 1890;Humphreys and Bruce, 1989) ( Yee et al, 2001;Cater et al, 2003;Mastoropoulou et al, 2005a;Mastoropoulou, 2006;Chalmers et al, 2006;Longhurst et al, 2006) inattentional blindness (Rock et al, 1992;Mack and Rock, 1998;Simons and Chabris, 1999) (Cater et al, 2003;Mastoropoulou et al, 2005a;Mastoropoulou, 2006) modality appropriateness hypothesis (Howard and Templeton, 1966;Welch and Warren, 1980) (Mastoropoulou and Chalmers, 2004;Mastoropoulou, 2006;Hulusic et al, 2009Hulusic et al, , 2010a auditory driving effect (Gebhard and Mowbray, 1959;Shipley, 1964;Wada et al, 2003;Recanzone, 2003) (Mastoropoulou and Chalmers, 2004;Mastoropoulou, 2006;Hulusic et al, 2009Hulusic et al, , 2010a temporal ventriloquism (Morein-Zamir et al, 2003;Bertelson and Aschersleben, 2003;Aschersleben and Bertelson, 2003;Burr et al, 2009) (Hulusic et al, 2009(Hulusic et al, , 2010a illusory flash induced by sound (Shams et al, 2000(Shams et al, , 2002) …”