Immersive Video Coding: Should Geometry Information Be Transmitted as Depth Maps?

Abstract

Immersive video often refers to multiple views with texture and scene geometry information, from which different viewports can be synthesized on the client side. To design efficient immersive video coding solutions, it is desirable to minimize bitrate, pixel rate and complexity. We investigate whether the classical approach of sending the geometry of a scene as depth maps is appropriate to serve this purpose. Previous work shows that bypassing depth transmission entirely and estimating depth at the client side improves the synthesis performance while saving bitrate and pixel rate. In order to understand if the encoder side depth maps contain information that is beneficial to be transmitted, we first explore a hybrid approach which enables partial depth map transmission using a block-based RD-based decision in the depth coding process. This approach reveals that partial depth map transmission may improve the rendering performance but does not present a good compromise in terms of compression efficiency. This led us to address the remaining drawbacks of decoder side depth estimation: complexity and depth map inaccuracy. We propose a novel system that takes advantage of high quality depth maps at the server side by encoding them into lightweight features that support the depth estimator at the client side. These features allow reducing the amount of data that has to be handled during decoder side depth estimation by 88%, which significantly speeds up the cost computation and the energy minimization of the depth estimator. Furthermore, -46.0% and -37.9% average synthesis BD-Rate gains are achieved compared to the classical approach with depth maps estimated at the encoder.

Recursive block splitting in feature‐driven decoder‐side depth estimation

“…State-of-the-art approaches [12,21] split the block only once. Therefore, they cannot fit the more complicated structure of the depth map.…”

“…A different approach is shown in feature-driven depth estimation [12], which assumes three different features already within the encoded bitstream. The first type is the depth range within each block of input depth maps (Figure 1).…”

“…The size of blocks is usually not fixed. For example, the blocks can be divided into four square subblocks [12] (Figure 2) or two rectangular subblocks (earlier proposed and evaluated by Szydełko and others [20]). All of these division types are shown in Figure 3.…”

“…The results presented in the previous subsection were averaged over all QPs to obtain a single PSNR value for F I G U R E 8 Bitrate of extracted features (averaged over all sequences and views) versus PSNR of synthesized views (averaged over all sequences, views, and QPs) for 18 tested configurations; state-of-the-art approaches: Q-128-1 and Q-64-1 [12] and A-128-1 and A-64-1 [21] F I G U R E 9 Decoding versus PSNR of synthesized views (both values are averaged over all sequences, views, and QPs) for 18 tested configurations; state-of-the-art approaches: Q-128-1 and Q-64-1 [12] and A-128-1 and A-64-1 [21] F I G U E 1 0 Bitrate of extracted features (averaged sequences and views) versus decoding time (averaged over all sequences, views, and QPs) for 18 tested configurations; state-ofthe-art approaches: Q-128-1 and Q-64-1 [12] and A-128-1 and A-64-1 [21] each tested configuration. However, as mentioned in Section 4.1, VVC encoding was repeated five times with different values of QP to allow the analysis of the RD curves.…”

“…However, in most cases, the geometry information (eg., in the form of a depth map for each view [11]) is available on the encoder side. This situation enables the encoder to send a set of features that could make it easier to estimate geometry in the decoder [12,13]. These features include block-wise information, such as a range of depth in a block or a skip flag that indicates that the depth in a block did not change from the previous frame.…”