Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures

Abstract

Abstract-While much of multiview video coding focuses on the rate-distortion performance of compressing all frames of all views for storage or non-interactive video delivery over networks, we address the problem of designing a frame structure to enable interactive multiview streaming, where clients can interactively switch views during video playback. Thus, as a client is playing back successive frames (in time) for a given view, it can send a request to the server to switch to a different view while continuing uninterrupted temporal playback. Noting that standard tools for random access (i.e., I-frame insertion) can be bandwidth-inefficient for this application, we propose a redundant representation of I-, P-, and "merge" frames, where each original picture can be encoded into multiple versions, appropriately trading off expected transmission rate with storage, to facilitate view switching. We first present ad hoc frame structures with good performance when the view-switching probabilities are either very large or very small. We then present optimization algorithms that generate more general frame structures with better overall performance for the general case. We show in our experiments that we can generate redundant frame structures offering a range of tradeoff points between transmission and storage, e.g., outperforming simple I-frame insertion structures by up to 45% in terms of bandwidth efficiency at twice the storage cost.Index Terms-Media interaction, multiview video coding, video streaming.

Coding and replication co-design for interactive multiview video streaming

“…Though suitable for compact storage of all multiview data (e.g., on a DVD disc), for IMVS application [2] where only a single requested view per client is needed at one time, complicated inter-frame dependencies among coded frames across time and view reduce the random decodability of the video stream. In contrast, we propose an efficient frame structure where an image can be encoded into multiple versions, so that the appropriate version can be transmitted depending on the available content in decoder's buffer at stream time, in order to reduce server transmission rate.…”

“…In interactive multiview video streaming (IMVS) services [2], a client can play back the captured multiview video content in time in a single view, and may switch at any time to new views in order to observe a scene of interest from different viewing angles. Active selection of viewpoints by a client can engender a depth perception in the observed 3D scene [3].…”

“…First, in order to facilitate view-switching and storage, we propose a coding structure based on redundant P-frames and distributed source coding (DSC) frames to help users navigate among adjacent views [2], [5]. Given this novel structure, we then propose an efficient and cooperative heuristic called MVCR (Multiview Video Cooperative Replication) to efficiently replicate views.…”

Incentive analysis for cooperative distribution of interactive multiview video

“…In new interactive multiview view streaming (IMVS) services [1], a client can periodically select one out of many captured views available for observation as the video is played back in time. In response, server sends only preencoded data for the single requested view (rather than all the captured views) to lower streaming rate.…”

“…In the literature, [1] designed frame structures using distributed source coding (DSC) [4] for IMVS to achieve bandwidth-efficient view switching. [5] used DSC for both view-switching and cooperative packet loss recovery in a WWAN multiview video multicast system.…”