Effect of compressed offline foveated video on viewing behavior and subjective quality

Abstract

Offline foveation is a technique to improve the compression efficiency of digitized video. The general idea behind offline foveation is to blur video regions where no or a small number of previewers look without decreasing the subjective quality for later viewers. It relies on the fact that peripheral vision is reduced compared to central vision, and the observation that during free-viewing humans' gaze positions generally coincide when watching video. In this article, we conduct two experiments to assess how offline foveation affects viewing behavior and subjective quality. In the first experiment, 15 subjects free-viewed six video clips before and after offline foveation whereas in the second experiment we had 17 subjects assessing the quality of these videos after one, two, and three consecutive viewings. Eye movements were measured during the experiments. Results showed that, although offline foveation prior to encoding with H.264 yielded data reductions up to 52% (20% average) on the tested videos, it had little or no effect on where people looked, their intersubject dispersion, fixation duration, saccade amplitude, or the experienced quality during first-time viewing. However, seeing the videos more than once increased the intersubject dispersion and decreased the subjective quality. In view of these results, we discuss the usage of offline foveated video in practical applications.

Subjective Quality Evaluation of Foveated Video Coding Using Audio-Visual Focus of Attention

“…Also, the eye tracking experiment presented in [43] showed that gaze fixation locations are not significantly altered by visible coding artifacts. Nevertheless, the impairment in the background region of the foveated sequences used in Experiment 2 are extremely severe in comparison to those used in [37], [43]. Additionally, the artifacts in [37] and [43] were uniformly distributed, while in our case they were not.…”

“…These observations may raise a question about the effectiveness of the foveated coding. However, it should be noted that the results in [37], [42] and ours were drawn from different conditions, i.e., absence and existence of the sound signal. In our experiment, the memory effect was not clearly observed for the speech contents, which might be because a talking face accompanied with speech was interesting enough even in the second viewing.…”

“…This could be due to the fact that the strong artifacts in the background region of the foveated image frames drew the attention of the observers. In [37], it was shown that foveation does not change significantly the gaze pattern of human observers. Also, the eye tracking experiment presented in [43] showed that gaze fixation locations are not significantly altered by visible coding artifacts.…”

“…In [37], repeated viewings of the same stimuli cause changes in gaze patterns of observers, so that the background region having poor quality tends to be attended after multiple viewing. The work in [42] also showed that perceptual memory leads to increases in the impact of top-down influences on attentional selection during natural vision.…”

“…It is worth mentioning that, in some prior work, offline foveation (e.g., blurring) was performed and then the foveated image frames were encoded, where resulting coding gains should be considered as lower bounds of the expected gains [2], [35], [37]. In contrast, the foveation process of this method is directly embedded in the H.264/AVC encoding by using the FMO scheme.…”