Cooperative Tile-Based 360° Panoramic Streaming in Heterogeneous Networks Using Scalable Video Coding

Zhang, Xiaoyi; Hu, Xinjue; Zhong, L.; Shirmohammadi, Shervin; Zhang, Lin

doi:10.1109/tcsvt.2018.2886805

Cited by 27 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This content preparation and delivery may introduce prohibitive delays, as the playback delays are very short for live streaming of 360-degree video. An alternative to DASH-like video streaming is the use of scalable video coding (SVC) [ 124 , 125 , 126 ]. In SVC-based systems, the video is encoded multiple quality layers (a base layer and possibly multiple enhancement layers).…”

Section: Learning-based Transmissionmentioning

confidence: 99%

Machine Learning for Multimedia Communications

Thomos

Maugey

Toni

2022

Sensors

View full text Add to dashboard Cite

Machine learning is revolutionizing the way multimedia information is processed and transmitted to users. After intensive and powerful training, some impressive efficiency/accuracy improvements have been made all over the transmission pipeline. For example, the high model capacity of the learning-based architectures enables us to accurately model the image and video behavior such that tremendous compression gains can be achieved. Similarly, error concealment, streaming strategy or even user perception modeling have widely benefited from the recent learning-oriented developments. However, learning-based algorithms often imply drastic changes to the way data are represented or consumed, meaning that the overall pipeline can be affected even though a subpart of it is optimized. In this paper, we review the recent major advances that have been proposed all across the transmission chain, and we discuss their potential impact and the research challenges that they raise.

show abstract

Section: Learning-based Transmissionmentioning

confidence: 99%

Machine Learning for Multimedia Communications

Thomos

Maugey

Toni

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Users can adaptively fetch tiles of different quality according to their network conditions. A few research, like [8], choose to explore the cooperate chance between several VR users that can minimize the bandwidth consumption. Most still choose to dedicate to single user streaming.…”

Section: Related Workmentioning

confidence: 99%

Reinforcement Learning Driven Adaptive VR Streaming with Optical Flow Based QoE

Quan

Pan

Xiang

et al. 2020

2020 IEEE 18th International Conference on Industrial Informatics (INDIN)

Self Cite

View full text Add to dashboard Cite

With the merit of containing full panoramic content in one camera, Virtual Reality (VR) and 360 • videos have attracted more and more attention in the field of industrial cloud manufacturing and training. Industrial Internet of Things (IoT), where many VR terminals needed to be online at the same time, can hardly guarantee VR's bandwidth requirement. However, by making use of users' quality of experience (QoE) awareness factors, including the relative moving speed and depth difference between the viewpoint and other content, bandwidth consumption can be reduced. In this paper, we propose OFB-VR (Optical Flow Based VR), an interactive method of VR streaming that can make use of VR users' QoE awareness to ease the bandwidth pressure. The Just-Noticeable Difference through Optical Flow Estimation (JND-OFE) is explored to quantify users' awareness of quality distortion in 360 • videos. Accordingly, a novel 360 • videos QoE metric based on PSNR and JND-OFE (PSNR-OF) is proposed. With the help of PSNR-OF, OFB-VR proposes a versatile-size tiling scheme to lessen the tiling overhead. A Reinforcement Learning(RL) method is implemented to make use of historical data to perform Adaptive BitRate(ABR). For evaluation, we take two prior VR streaming schemes, Pano and Plato, as baselines. Vast evaluations show that our system can increase the mean PSNR-OF score by 9.5-15.8% while maintaining the same rebuffer ratio compared with Pano and Plato in a fluctuate LTE bandwidth dataset. Evaluation results show that OFB-VR is a promising prototype for actual interactive industrial VR. A prototype of OFB-VR can be found in https://github.com/buptexplorers/OFB-VR.

show abstract

“…A variant of such approach uses multiple layers combined with the Adaptive Resolution Change (ARC) mechanism of the VVC standard [6]. The tile splitting approach encodes each ERP frame as a set of independent nonoverlapping tiles of fixed-size [7][8][9][10]. Such encoding mechanism allows receivers to select and decode any tile according to the viewing area (FoV or relevant viewport), i.e., enabling partial decoding of the full FoV.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, in [11] using the cube-map projection, each cube face is split into multiple tiles and multiple layers for resolution adaptation. A similar scheme was also used in [10] for cooperative streaming of 360°v ideo.…”

Section: Introductionmentioning

confidence: 99%

Attention-driven tile splitting method for improved efficiency of omnidirectional versatile video coding

Carreira

Faria

Tavora

et al. 2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

A common approach used in omnidirectional video coding is based on frame splitting into tiles, allowing partial delivery of only the subset of tiles that is necessary to render the user's current viewing region, defined as a specific viewport or Field-of-View (FoV). Since tiles can be independently encoded, such mechanism provides a flexible solution for encoding planar representations with ultra-high definition (UHD), such as the Equirectangular Projection (ERP), using Versatile Video Coding (VVC). By only selecting and transmitting the coded data that is required to render the necessary FoV, rather than the full 360°, a great deal of bandwidth can be saved. While current solutions are based on splitting the omnidirectional video frames into tiles of equal size, this paper proposes a new approach based on adaptive tile size, driven by visual attention. Those regions where the visual attention is higher are partitioned in smaller tiles to obtain higher bit rate granularity, allowing to decode the most frequent FoVs with minimum out-of-FoV pixels and reduced bandwidth. Optimal tile boundaries are found by solving a lagrangian minimisation problem with a cost function that achieves the best tradeoff between the standard deviation and the average attention-weighted bit rate per tile. The experimental results show that an average of 7.17% and 17.73% of bit rate savings is obtained in comparison with conventional tilling methods for the commonly used FoVs of 90°×90°and 45°×45°, respectively.

show abstract

Cooperative Tile-Based 360° Panoramic Streaming in Heterogeneous Networks Using Scalable Video Coding

Cited by 27 publications

References 17 publications

Machine Learning for Multimedia Communications

Machine Learning for Multimedia Communications

Reinforcement Learning Driven Adaptive VR Streaming with Optical Flow Based QoE

Attention-driven tile splitting method for improved efficiency of omnidirectional versatile video coding

Contact Info

Product

Resources

About