Coding structure optimization for interactive multiview streaming in virtual world observation

Abstract

Abstract-While most multiview coding techniques focus on compressing all frames in a multiview video sequence in a ratedistortion optimal manner, in this paper we address the problem of interactive multiview streaming, where we minimize the expected transmission rate of an interactive multiview video stream, where the observer can select the view of the next frame, subject to a storage constraint. We show that gains can be achieved by optimizing the trade-off between overall storage and transmission rate, i.e., by storing a more redundant multiview representation (where some frames are encoded more than once, each time using a different reference frame) it is possible to reduce the overall bandwidth needed for online interactive viewing. We show that our proposed redundant representation can reduce the transmission cost of interactive multiview streaming by up to 65% as compared to a good non-redundant representation for the same storage constraint.

Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures

“…In our previous work, we formally posed the IMVS problem as a combinatorial optimization in [2], proved its NP-hardness, and provided two heuristics-based algorithms to find good frame structures while allowing unlimited rerouting for IMVS. [25] is a more thorough and analytical treatment of the same problem with limited rerouting, using only I-and P-frames in the structure.…”

“…Though (8) differs from the definition in [2], a similar proof can be easily constructed to show that is NP-hard. Given the computational difficulty of (8), we focus next on solving the corresponding Lagrangian optimization for given Lagrange multiplier instead 10 :…”

“…We define as a partial structure constructed from up till switching instant is simply . For given partial structure , each frame in has a display probability computable front-to-back using (2). Each frame will switch to view with probability either via a P-frame predicted from , or via an M-frame , each with different local Lagrangian costs at instant .…”

“…The cameras can be parts of a real physical camera system [1] or a virtual camera setup inside a computer [2] using API manipulation of graphics cards [3]. Much of the previous research in multiview video focuses on compression: to design coding techniques exploiting temporal (across time) and spatial (across view) correlation to encode all frames of all views of a multiview sequence in a rate-distortion optimal manner [4]- [7].…”

“…1(b). 2 Note, however, that all implementations of M-frame must have larger transmission rate than a P-frame, since by definition, an M-frame must merge multiple (more than one) decoding paths each with a different predictor-a more stringent reconstruction requirement than a P-frame which has only one decoding path with one known predictor. If we encode an M-frame using the most storage-efficient implementation available-the left-most point on the convex hull 3 -for every view at every switching point, this leads to the most storage-efficient frame structure.…”