Viewport-adaptive 360-degree video coding

Deep Learning and Bidirectional Optical Flow Based Viewport Predictions for 360° Video Coding

“…Table 5. illustrates the performance comparison between the state-of-the-art [22], [42], [44] 1 works and the VPred-SCNN + LQAOSCNN. It is evident that the proposed variant outperforms the state-of-the-art methods in terms of perceptual compression efficiency, however it remains computationally complex.…”

“…Aforementioned state-of-the-art researches, perceptually suffer losses resulting from the inaccurate prediction of QERs in the ERP 2 . Furthermore, other than Hu et al [44] 3 , both Sreedhar et al [42] and TSP [22] show inconsistent gains and heavy losses (up to 10.15 dB and 12.52 dB respectively) in viewport quality due to the application of re-sampling and re-packaging of poorly predicted QERs in their methodologies. Moreover, the 35% compression performance achieved for CourgarsTreats sequences by TSP [22] comes at perceptual quality loss of 5.47dB.…”

“…The re-sampling strategies in Sreedhar et al [42] and TSP [22] are cable of exploiting the fine distribution of viewport coordinates as opposed to the coarse distributions, which explains the variations in their respective perceptual compression performances and quality losses. Also, the idea of viewport dependent coding parameter adaptation at CTU level followed by Hu et al [44] is not an efficient strategy for VVC due its finer QTBT+multi tree type partitioning structure. Moreover, for a better understanding, a visual comparison of a given viewport generated from the reference, anchor, state-of-the-art works and VPredSCNN + LOAQSCNN with respect to ManHattan sequence is shown in FIGURE 10.…”

“…In contrast, Facebook developed Barrel layout based on AI-driven saliency maps to identify user interested regions prior to encoding [43]. Furthermore, Hu et al [44] reports learned weights driven viewport dependent Lagrange optimization and adaptive quantization techniques at Coding Tree Unit (CTU) level which improves the perceptual compression gains of HEVC by 26%.…”

“…Building on the non-tiled approach for 360 • video coding, this paper proposes several novel approaches that differ from the state-of-the-art techniques [22], [42]- [44], in number of ways. Firstly, the proposed research applies two different techniques for viewport prediction; a hybrid approach that combines bi-directional optical flow estimation and a Gaussian filtering technique to extract saliency features in both spatial and temporal domains and an Spherical CNN incorporated deep learning approach to obtain spherical features only in the spatial domain.…”