A real-time interactive multi-view video system

Ortega

IEEE Trans. on Image Process.

2011

129

Abstract-While much of multiview video coding focuses on the rate-distortion performance of compressing all frames of all views for storage or non-interactive video delivery over networks, we address the problem of designing a frame structure to enable interactive multiview streaming, where clients can interactively switch views during video playback. Thus, as a client is playing back successive frames (in time) for a given view, it can send a request to the server to switch to a different view while continuing uninterrupted temporal playback. Noting that standard tools for random access (i.e., I-frame insertion) can be bandwidth-inefficient for this application, we propose a redundant representation of I-, P-, and "merge" frames, where each original picture can be encoded into multiple versions, appropriately trading off expected transmission rate with storage, to facilitate view switching. We first present ad hoc frame structures with good performance when the view-switching probabilities are either very large or very small. We then present optimization algorithms that generate more general frame structures with better overall performance for the general case. We show in our experiments that we can generate redundant frame structures offering a range of tradeoff points between transmission and storage, e.g., outperforming simple I-frame insertion structures by up to 45% in terms of bandwidth efficiency at twice the storage cost.Index Terms-Media interaction, multiview video coding, video streaming.

Section: Interactive Multiview Video Streamingmentioning

confidence: 99%

Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures

Ortega

IEEE Trans. on Image Process.

2011

129

“…The most similar work to our IMVS work is [8], which developed three separate frame structures to support three types of interactivity: view switching, frozen moment and view sweeping. While the authors recognized the importance of a "proper tradeoff among flexibility (interactivity), latency and bandwidth cost", no explicit optimization was performed to find the best tradeoffs of these quantities in one structure.…”

Section: B Interactive Multiview Video Streamingmentioning

confidence: 99%

Delay-Cognizant Interactive Streaming of Multiview Video With Free Viewpoint Synthesis

Xiu

Liang

2012

IEEE Trans. Multimedia

In interactive multiview video streaming (IMVS), a client receives and observes one of many available viewpoints of the same scene, and periodically requests from server view-switches to neighboring views, as the video is played back in time uninterruptedly. One key technical challenge is to design a frame coding structure that facilitates periodic view-switching, and achieves an optimal tradeoff between storage cost and expected transmission rate. In this paper, we first propose three significant improvements over existing IMVS system, and then study the corresponding frame structure optimization. First, using depth-image-based rendering, the new IMVS system enables free viewpoint switching, i.e., by encoding and transmitting both texture and depth maps of captured views, a client can select and synthesize any virtual view from an almost continuum of viewpoints between the left-most and right-most captured views. Second, IMVS system adopts a more realistic Markovian view-switching model with memory that more accurately captures user behaviors than previous memoryless models. View-switching model is used in predicting client's future view-switching patterns. Third, assuming that the round-trip-time (RTT) delay during server-client communication is non-negligible, during an IMVS session, IMVS system additionally transmits redundant frames RTT into future playback, so that zero-delay view-switching can be achieved. Given these improvements, we formalize a new join optimization of the frame coding structure, transmission schedule, and quantization parameters of the texture and depth maps of multiple camera views. We propose an iterative algorithm to achieve fast and near-optimal solutions.The convergence of the algorithm is also demonstrated. Experimental results show that the proposed optimized rate allocation method requires 38% less transmission rate than the fixed rate allocation scheme. In addition, with the same storage, the transmission rate of the optimized frame structure can be up to 55% lower than that of I-frame-only structure, and 27% lower than that of the structure without distributed source coding (DSC) frames. Index Termsmultiview video, video streaming, media interaction, view synthesis

2012 Proceedings IEEE INFOCOM

“…At the first frame or head of a coding unit, a user can interactively select different views of the same time instant (first frames of other coding units) before deciding on one view to resume video playback. Because there is no time progression during this interactivity, we call this navigation of views of the same time instant static viewswitching (called frozen moment in [11]). See Fig.…”

Section: Imvs System and Coding Structurementioning

confidence: 99%

Coding and replication co-design for interactive multiview video streaming

Huang

Chan

et al. 2012

Abstract-Multiview video refers to the simultaneous capturing of multiple video views with an array of closely spaced cameras. In an interactive multiview video streaming (IMVS) system, a client can play back the content in time in a single view, and may observe a scene of interest by switching to different viewpoints. Users independently choose their own view navigation paths through the high-dimensional multiview data. Distributed servers are deployed to collaboratively replicate video content in order to support user scalability. Such a system typically presents challenges in both coding and content replication. In coding, the multiview video must be encoded in order to support efficient viewswitching and distributed replication. In content replication, it is important to decide which data blocks to store at each server to facilitate view-switches at any time.In this paper, we co-design a coding structure and a distributed content replication strategy. First, we propose a coding structure based on redundant P-frames and distributed source coding (DSC) frames to achieve efficiency in coding, view switches and content replication. We then propose a heuristic-based distributed and cooperative replication strategy to take advantage of the correlation between the multiple views for resource-effective content delivery. Simulation results show that our coding and replication codesign is cost-effective in supporting IMVS services.