Perceptual model for adaptive local shading and refresh rate

Abstract

Intel 2019]. All these works propose a dynamic quality control mechanism that allocates the rendering budget to those aspects of an image or animation that have the highest impact on the overall quality. In this work, we propose to control both the VRS state map and the refresh rate, based on all major factors affecting image quality: texture content, on-screen velocities, luminance, effective resolution, and display persistence. We build on the work of Denes et al. [2020], and extend the visual quality model to account for the important effect of texture content and VRS resolution. In contrast to that work, we offer local, rather than global, control of the resolution via VRS without the need for eye tracking.The key component of our Adaptive Local Shading and Refresh Rate (ALSaRR) method is a new Content-adaptive Metric of Judder, Aliasing and Blur (CaMoJAB) (Section 4). The metric is based on psychophysical models of contrast sensitivity with only a few parameters fitted to the data. We calibrate and validate our metric on various existing datasets as well as our new dataset collected by conducting a subjective quality experiment. The experiment measures the perceived loss of quality due to shading rate reduction under a large range of display refresh rates, resolutions, display persistence, luminance, contrast, and content velocity (Section 4.3). Our ALSaRR method uses the new metric to create per-texture quality functions, which are used for an approximate solution of the knapsack problem: maximize perceived quality for a given rendering budget (Section 5). Unlike the method of Yang et al. [2019], which controls VRS to avoid any visual loss regardless of the per-frame rendering cost, our goal is to find the best trade-off of spatio-temporal resolution under a limited rendering budget.The main contributions of our work 1 are:• Content-adaptive Metric of Judder, Aliasing and Blur (CaMo-JAB), derived from psychophysical models and calibrated on several datasets. • A dataset of motion quality for animations rendered with different shading rates, motion velocities, textures, refresh rates and display angular resolution (in pixels-per-degree), and persistence. • Adaptive Local Shading and Refresh Rate (ALSaRR) method for control of real-time rendering, which maximizes the quality of animation under a limited budget.

Foveated rendering: A state-of-the-art survey

“…This reprojection detected and reevaluated artifacts and disocclusions according to a confidence value determined by a perception-based metric. Jindal et al [154] proposed the variable-rate shading pipeline to accelerate rasterization rendering performance. This approach divides the output image into a number of 16×16 image tiles, and subsequently adaptively adjusts the shading accuracy and refresh rate of each image tile based on spatio-temporal and the spatio-luminance CSFs.…”

Image Features Influence Reaction Time: A Learned Probabilistic Perceptual Model for Saccade Latency

“…We leave the task of creating a multiband and color-aware model tailored for measuring reactive latency for future work. Our model also does not consider scenarios where motion [Jindal et al 2021] and refresh rates [Krajancich et al 2021] may play a critical role, especially during saccades [Schweitzer and Rolfs 2021]. We aim to extend our model to consider spatio-temporal effects and complex dynamic scenarios in the future.…”

18‐1: Invited Paper: Modeling and Optimizing Human‐in‐the‐Loop Visual Perception Using Immersive Displays: A Review

“…fovea, and decreasing progressively toward peripheral vision. Several recent approaches in the area of foveated rendering have proposed to utilize this effect to improve performance and quality of graphics on near-eye displays [12,13,20,28,30]. An example is shown in Figure 2(a).…”

“…An example is shown in Figure 2(a). However, due to the complexity of foveated rendering algorithms, realizing practical performance enhancement in FPS demands novel GPU architectural support [30].…”

“…Consequently, high refresh-rate displays on desktop, mobile as well as VR devices are becoming increasingly common [35]. However, rendering highquality visual content at ultra-high framerates is challenging, and has led to explorations of variable and dynamic refresh rate systems [30,34]. Further, compared to spatial models, computational models of how humans perceive temporal changes in images are relatively new and underdeveloped [26,27].…”