A Survey on Perceptually Optimized Video Coding

Abstract

Videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG) and high fidelity, which provide users with more realistic visual experiences. However, the amount of video data increases exponentially and requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to exploit visual redundancies in videos so as to maximize compression efficiency. In this paper, we present a systematic survey on the recent advances and challenges on perceptually optimized video coding. Firstly, we present problem formulation and framework of perceptually optimized video coding, which includes visual perception modelling, visual quality assessment and perception guided coding optimization. Secondly, the recent advances on visual factors, key computational visual models and quality assessment models are presented. Thirdly, we do systematic review on perceptual video coding optimizations from four key aspects, which includes perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages and challenges are presented. Fourthly, perceptual coding performance of latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities on perceptual video coding are identified.CCS Concepts: • Computing methodologies → Image compression.

Domain-Specific Fusion Of Objective Video Quality Metrics

“…1, where BD-rates obtained for different encoders (and with and without the use of deep perceptual preprocessing [4]) vary significantly from one metric to another. Such observations have been reported for a variety of applications and test conditions by a number of independent studies [2,37]. This is further exasperated with the use of generative neural network loss functions [6,17].…”

“…While it is well known that bitrate efficiency increases with the use of encoding presets of increased complexity, e.g., x264 AVC preset=veryslow versus preset=fast, it is not clear what is the relationship between different encoders. The use of state-of-the-art preprocessing technologies for perceptual optimization [4,10,20,37] complicates this further, as such preprocessing makes the encoding even more perceptually-tuned than the original perceptual tuning within the encoders & recipes themselves. The baseline results corresponding to this experiment are shown in the left four plots of Fig.…”

“…However, while such metrics have been evolving, so have algorithms that focus on perceptual quality, such as psychovisuallyoptimized video coding tools within AV1 and VVC [10,20,35,37], deep perceptual preprocessing for video coding [4,37], perceptuallyoptimized video denoising and upscaling [6] and synthetic video generation with generative adversarial networks [17]. Given that many of these systems optimize for variations of cost functions that align to components of widely-used metrics like VMAF and SSIM, this makes their performance diverge when assessed with multiple state-of-the-art metrics.…”