Merge Frame Design for Video Stream Switching Using Piecewise Constant Functions

Abstract

Abstract-The ability to efficiently switch from one preencoded video stream to another (e.g., for bitrate adaptation or view switching) is important for many interactive streaming applications. Recently, stream-switching mechanisms based on distributed source coding (DSC) have been proposed. In order to reduce the overall transmission rate, these approaches provide a "merge" mechanism, where information is sent to the decoder such that the exact same frame can be reconstructed given that any one of a known set of side information (SI) frames is available at the decoder (e.g., each SI frame may correspond to a different stream from which we are switching). However, the use of bitplane coding and channel coding in many DSC approaches leads to complex coding and decoding. In this paper, we propose an alternative approach for merging multiple SI frames, using a piecewise constant (PWC) function as the merge operator. In our approach, for each block to be reconstructed, a series of parameters of these PWC merge functions are transmitted in order to guarantee identical reconstruction given the known side information blocks. We consider two different scenarios. In the first case, a target frame is first given, and then merge parameters are chosen so that this frame can be reconstructed exactly at the decoder. In contrast, in the second scenario, the reconstructed frame and merge parameters are jointly optimized to meet a rate-distortion criteria. Experiments show that for both scenarios, our proposed merge techniques can outperform both a recent approach based on DSC and the SP-frame approach in H.264, in terms of compression efficiency and decoder complexity.

Landmarking for Navigational Streaming of Stored High-Dimensional Media

“…We next iteratively add/remove inter-coded MDUs using a fast branch-and-bound (B&B) method [18] to refine the MDU structure. Taking interactive LF images and viewport adaptive 360 • images as illustrative applications, and using I-, P-and previously proposed merge (M-) frames [19] to encode MDUs, we show experimentally that landmark-based MDU structures can noticeably reduce 2 By coding cost, we mean the coding bitrate, not coding complexity.…”

“…Two recent studies [7] [9] focused on the use of distributed source coding (DSC) frames [26] and merge frames [19] for view-switching without coding drift. [8] discussed the general notion of landmark to be used in ILFS but did not provide a formal optimization framework.…”

“…We encode one viewport-worth of data as one MDU, thus achieving good coding efficiency due to intra-prediction. At the same time, we achieve low view-switching cost thanks to the inter-coding of neighboring MDUs using advanced DSC tools [19] and landmarks.…”

“…To accomplish this, a merge-coded MDU (M-MDU) M j is employed to "merge" different reconstructions of P j (i) identically to I-MDU I j . Many previous works have studied the merge operator, including SP-frame in H.264 [28], distributed source coding (DSC) in [26], and the merge frame in [19]. We discuss detailed implementation of our M-MDU based on [19] in Section VII-A.…”

Multi-stream switching for interactive virtual reality video streaming

“…A receiving user can periodically request switches to neighboring camera views, and the server in response switches video streams with minimum discruption to the user's viewing experience. To facilitate stream-switching, new frames like DSC frame [9] and merge frame (M-frame) [10] were proposed. Unlike IMVS [4][5][6][7][8], we optimize the division of 360 VR video into multiple streams covering different view ranges given a constant RTT.…”