Video processing for panoramic streaming using HEVC and its scalable extensions

Fuente, Yago Sanchez de la; Skupin, Robert; Schierl, Thomas

doi:10.1007/s11042-016-4097-4

Cited by 22 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, while watching a specific view and a switch to another view is necessary, that other view would need to be decoded. This scheme is similar to conventional adaptive streaming techniques [23,24]. However, the absence of inter-view dependencies leaves a large room for improvement in terms of redundancy exploitation, and therefore compression efficiency.…”

Section: Interframementioning

confidence: 93%

Random access prediction structures for light field video coding with MV-HEVC

Avramelos

Praeter

Wallendael

et al. 2020

Multimed Tools Appl

View full text Add to dashboard Cite

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, interview and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multiview video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access. Keywords Light field video coding • multi-view video coding • MV-HEVC • prediction structures • random access • virtual reality • free navigation

show abstract

Section: Interframementioning

confidence: 93%

Random access prediction structures for light field video coding with MV-HEVC

Avramelos

Praeter

Wallendael

et al. 2020

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…The tile-based video streaming method is a highly efficient technique for VR multimedia systems with HMD, because it transmits only the video data in the user's viewport. Therefore, compared to the conventional approach that transmits the entire pictures, a tile-based approach can achieve higher video quality at limited network bandwidth [28].…”

Section: Saliency Map-based Bit-rate Adaptation For Vr Streaming By Sungkyunkwan Universitymentioning

confidence: 99%

Overview of Motion-to-Photon Latency Reduction for Mitigating VR Sickness

2021

KSII TIIS

View full text Add to dashboard Cite

For several years, virtual reality (VR) and augmented reality (AR) technologies have been improving. However, some hurdles remain that slow down the distribution of VR and AR devices, such as head-mounted display (HMD), and related consumer content. One issue is VR motion sickness, which has been experienced by users using 360 degree VR content via HMD. This paper discusses the related international standardization work that classifies the factors causing VR sickness, and proposes the process for VR sickness level evaluation.Among the factors causing VR sickness, many research institutes regard minimizing MTP (Motion-to-Photon) latency as the key enabler to mitigate VR sickness. Thus, this paper introduces research trends of MTP latency measurement and MTP latency mitigation. This paper categorizes the research on MTP latency measurement into 2 categories of hardware-based approach and software code-level approach. The 2 approaches have different pros and cons depending on use-case, purpose, and architecture of each multimedia system. The pros and cons are addressed in this paper. Additionally, the research on mitigating MTP latency with diverse strategies such as proactive computing, caching, and edge server technology is explained, and compared to conventional technologies, shows improved performance.

show abstract

“…As can be seen in Figure 2, in each workflow, one HM scanpath is generated through the interaction between the FoV extractor 3 and HM scanpath predictor. Specifically, {o n t } T t=1 denotes the FoVs of frames from 1 to T in the n-th DRL workflow.…”

Section: Framework Of Offline-dhpmentioning

confidence: 99%

“…Given the predicted HM, the eye fixations within the FoV can be further estimated using the conventional saliency detection methods [2] for 2D video. The same as traditional 2D video, the attention model can be extensively utilized in many areas of panoramic video, such as region-of-interest compression [3], visual quality assessment [4], [5], rendering [6], synopsis [7], and automatic cinematography [8].…”

Section: Introductionmentioning

confidence: 99%

Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Song

Wang

et al. 2019

IEEE Trans. Pattern Anal. Mach. Intell.

189

View full text Add to dashboard Cite

Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.

show abstract

Video processing for panoramic streaming using HEVC and its scalable extensions

Cited by 22 publications

References 17 publications

Random access prediction structures for light field video coding with MV-HEVC

Random access prediction structures for light field video coding with MV-HEVC

Overview of Motion-to-Photon Latency Reduction for Mitigating VR Sickness

Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Contact Info

Product

Resources

About