Attention-driven tile splitting method for improved efficiency of omnidirectional versatile video coding

Abstract

A common approach used in omnidirectional video coding is based on frame splitting into tiles, allowing partial delivery of only the subset of tiles that is necessary to render the user's current viewing region, defined as a specific viewport or Field-of-View (FoV). Since tiles can be independently encoded, such mechanism provides a flexible solution for encoding planar representations with ultra-high definition (UHD), such as the Equirectangular Projection (ERP), using Versatile Video Coding (VVC). By only selecting and transmitting the coded data that is required to render the necessary FoV, rather than the full 360°, a great deal of bandwidth can be saved. While current solutions are based on splitting the omnidirectional video frames into tiles of equal size, this paper proposes a new approach based on adaptive tile size, driven by visual attention. Those regions where the visual attention is higher are partitioned in smaller tiles to obtain higher bit rate granularity, allowing to decode the most frequent FoVs with minimum out-of-FoV pixels and reduced bandwidth. Optimal tile boundaries are found by solving a lagrangian minimisation problem with a cost function that achieves the best tradeoff between the standard deviation and the average attention-weighted bit rate per tile. The experimental results show that an average of 7.17% and 17.73% of bit rate savings is obtained in comparison with conventional tilling methods for the commonly used FoVs of 90°×90°and 45°×45°, respectively.

A Survey on Virtual Reality over Wireless Networks: Fundamentals, QoE, Enabling Technologies, Research Trends and Open Issues

“…Adaptive tiling based on the various requirements has drawn considerable attention from the research community. For instance, a content adaptive non-uniform tiling scheme proposed in [96], a visual attention-driven adaptive tile splitting method presented in [97], and a reinforcement learning (RL)-based rate adaptation with adaptive prediction and tiling investigated in [98] are some of these works. On the other hand, authors in [99] presented a scheme to determine the optimal tile size based on the content-specific characteristics and empirical distributions over user views of the video segments.…”