Interactive Multiview Video System With Low Complexity 2D Look Around at Decoder

Abstract

Abstract-Multiview video with interactive 2D look around at the receiver is a challenging application with several issues in terms of effective use of storage and bandwidth resources, reactivity of the system, quality of the viewing experience and system complexity. The impression of 3D immersion is highly dependent on the smoothness of the navigation and thus on the number of 2D viewpoints. The classical decoding system for generating virtual views first projects a reference or encoded frame to a given viewpoint and then fills in the holes due to potential occlusions. This last step still constitutes a complex operation with specific software or hardware at the receiver and requires a certain quantity of information from the neighboring frames for ensuring consistency between the virtual images. In this work we propose a new approach that shifts most of the burden due to interactivity from the decoder to the encoder, by anticipating the navigation of the decoder and sending auxiliary information that guarantees temporal and interview consistency. This leads to an additional cost in terms of transmission rate and storage, which we minimize by using optimization techniques based on the user behavior modeling. We show by experiments that the proposed system represents a valid solution for interactive multiview systems with classical decoders.Index Terms-2D look around viewing, interactivity, multiview video coding, view synthesis.

Convolutional Neural Network for Intermediate View Enhancement in Multiview Streaming

“…Moreover, the fixed prediction structure still limits the level of adaptivity. [20] anticipates the user's navigation pattern and sends auxiliary information that guarantees temporal and interview consistency. This method shifts the burden due to interactivity from the client side to the server side.…”

Adaptive multi-view video streaming using side information over peer-to-peer networks

“…The system optimizes the navigation quality experienced by the client using the bandwidth constraint. Also, the transmission of auxiliary information was studied in [27] to reduce the complexity of interactive multi-view video decoders. In the work, a residual frame was encoded at the server side by considering rate-distortion efficiency based on user behavior models.…”

Optimized Adaptive Streaming of Multi-video Stream Bundles

“…In summary, our results show that our system provides a good step in the right direction for providing seamless stream bundling services. [2], coding schemes that facilitate quick viewpoint switching [23], [24], [25], and that combine multiple texture/color and depth map streams to achieve free viewpoint streaming, including of virtual viewpoints [26], [5], [27], [28], [29]. In the context of 3D-TV, both multi-layered solutions [30] and prefetching information of alternative viewpoints [31] have been shown useful.…”