Random access prediction structures for light field video coding with MV-HEVC

Abstract

Computational imaging and light field technology promise to deliver the required six-degrees-of-freedom for natural scenes in virtual reality. Already existing extensions of standardized video coding formats, such as multi-view coding and multi-view plus depth, are the most conventional light field video coding solutions at the moment. The latest multi-view coding format, which is a direct extension of the high efficiency video coding (HEVC) standard, is called multi-view HEVC (or MV-HEVC). MV-HEVC treats each light field view as a separate video sequence, and uses syntax elements similar to standard HEVC for exploiting redundancies between neighboring views. To achieve this, interview and temporal prediction schemes are deployed with the aim to find the most optimal trade-off between coding performance and reconstruction quality. The number of possible prediction structures is unlimited and many of them are proposed in the literature. Although some of them are efficient in terms of compression ratio, they complicate random access due to the dependencies on previously decoded pixels or frames. Random access is an important feature in video delivery, and a crucial requirement in multiview video coding. In this work, we propose and compare different prediction structures for coding light field video using MV-HEVC with a focus on both compression efficiency and random accessibility. Experiments on three different short-baseline light field video sequences show the trade-off between bit-rate and distortion, as well as the average number of decoded views/frames, necessary for displaying any random frame at any time instance. The findings of this work indicate the most appropriate prediction structure depending on the available bandwidth and the required degree of random access. Keywords Light field video coding • multi-view video coding • MV-HEVC • prediction structures • random access • virtual reality • free navigation

Overview of MV-HEVC prediction structures for light field video

“…The four different prediction structures are named as Full, Khoury, Wang and Center, they can be found in the literature [12][13][14][15] and they are briefly described in Sec. 2.…”

“…The Full scheme was also used for comparison reasons in a previous work which was focusing on random access prediction schemes for light field video coding. 15 In that research work, a comparison between an all-intra scheme (full temporal and view random access), a fully-referenced scheme (increased number of inter-dependencies for improving compression rates but losing in terms of random access), a simulcast scheme (no inter-view prediction), and a scheme specifically tailored for applications where view random access is important. For the former scheme, namely Center, every frame of every view is predicted by the corresponding frame of the center tile/view.…”

“…12 Lastly, in previous work and within the context of this work, we proposed a three-directional scheme exploiting horizontal, vertical and diagonal correlations as well as a prediction scheme specifically designed to maximize view random access. 14,15 In this work, we focus on inter-view prediction and we evaluate and compare a selection of the above mentioned prediction schemes for light field video coding. Specifically, we validate the performance of those prediction schemes in terms of rate-distortion, and we examine the necessary trade-offs for gaining random access between different views (view random access).…”

“…Grid view of a 5 × 5 light field view ordering structure as presented within the context of this project. 14,15 On the left, a prediction scheme aiming towards view random access, and on the right, a scheme aiming towards compression efficiency. A similar referencing structure has been derived for the 8 × 8 case of this work…”

Downsampling Based Light Field Video Coding with Restoration Network Using Joint Spatio-Angular and Epipolar Information

“…1(a) and the LF video as 5D data (including time dimension). To ease the practical difficulty in communication and storage associated with their large data volume, their effective compression has become an active research topic producing many solutions [5]- [16] which can be divided into two categories based on their LF data format: lenslet video coding [7]- [12] and multi-view video coding [13]- [16].…”