Real-time foveation techniques for low bit rate video coding

Abstract

Lossy video compression methods often rely on modeling the abilities and limitations of the intended receiver, the Human Visual System (HVS), to achieve the highest possible compression with as little effect on perceived quality as possible. Foveation, which is nonuniform resolution perception of the visual stimulus by the HVS due to the non-uniform density of photoreceptor cells in the eye, has been demonstrated to be useful for reducing bit rates beyond the abilities of uniform resolution video coders. In this work, we present realtime foveation techniques for low bit rate video coding. First, we develop an approximate model for foveation. Then, we demonstrate that foveation, as described by this model, can be incorporated into standard motion compensation and Discrete Cosine Transform (DCT) based video coding techniques for low bit rate video coding, such as the H.263 or MPEG-4 video coding standards, without incurring prohibitive complexity overhead. We demonstrate that foveation in the DCT domain can actually result in computational speedups. The techniques presented can be implemented using the baseline modes in the video coding standards and do not require any modification to, or post processing at, the decoder.

Subjective Quality Evaluation of Foveated Video Coding Using Audio-Visual Focus of Attention

“…In the former approach, called offline foveation, the image frames to be encoded are processed in such a way that the quality of the peripheral region is degraded (e.g., low-pass filtering), which are inputted to an existing encoder (e.g., [2], [25]). The latter approach applies different encoding parameters for the foveated and peripheral regions (e.g., [4]), or allocates more bits on the foveated region in a rate-control scheme (e.g., [26]).…”

Bayesian Integration of Face and Low-Level Cues for Foveated Video Coding

“…Then the maximum normalized frequency detectable by the human eye, say f c (x t ), at a generic point x t = (x t ; y t ) of the frame can be calculated as [9] (see [31] for an in-depth discussion)…”

“…For instance, in terms of overhead relative to the uniform resolution baseline H.263 encoder, spatial domain foveation approximately introduces a 27% overhead, and DCT domain foveation a 1.5% overhead [31]. Summing up, we can expect that most of the time per frame will be spent in the optical flow stage, for what concerns the bottom-up computation; it is worth noting, that although pyramid construction is linear in time with the number of pixels, it requires a sufficient memory bandwidth, with special reference to the overcomplete representation of oriented pyramids (see [22], and [32] for detailed discussion).…”

Foveated multipoint videoconferencing at low bit rates

“…Foveation has been modeled for video coding purposes with a foveation cut-off frequency model that gives the largest frequency detectable by the HVS at a given eccentricity [5,6]. At any point on the display, a spatial frequency higher than the cut-off frequency is assumed to be imperceptible, and filtering it will not affect perceived quality.…”

“…In this paper, we propose using foveation for reducing the bandwidth requirements for MPVC over low bit rate networks. Foveation, which is non-uniform resolution perception of the visual stimulus by the Human Visual System (HVS) due to the nonuniform density of photoreceptor cells in the eye, has been demonstrated to be useful for low bit rate video coding using existing standards, and real-time algorithms for foveated video coding have been explored previously [5,6]. Foveated video coding improves the subjective quality at low bit rates, based on certain assumptions about the viewing configurations.…”