Optimal Representations for Adaptive Streaming in Interactive Multiview Video Systems

Abstract

Interactive multi-view video streaming (IMVS) services permit to remotely immerse within a 3D scene. This is possible by transmitting a set of reference camera views (anchor views), which are used by the clients to freely navigate in the scene and possibly synthesize additional viewpoints of interest. From a networking perspective, the big challenge in IMVS systems is to deliver to each client the best set of anchor views that maximizes the navigation quality, minimizes the view-switching delay and yet satisfies the network constraints. Integrating adaptive streaming solutions in free-viewpoint systems offers a promising solution to deploy IMVS in large and heterogeneous scenarios, as long as the multi-view video representations on the server are properly selected. We therefore propose to optimize the multi-view data at the server by minimizing the overall resource requirements, yet offering a good navigation quality to the different users. We propose a video representation set optimization for multiview adaptive streaming systems and we show that it is NP-hard. We therefore introduce the concept of multi-view navigation segment that permits to cast the video representation set selection as an integer linear programming problem with a bounded computational complexity. We then show that the proposed solution reduces the computational complexity while preserving optimality in most of the 3D scenes. We then provide simulation results for different classes of users and show the gain offered by an optimal multi-view video representation selection compared to recommended representation sets (e.g., Netflix and Apple ones) or to a baseline representation selection algorithm where the encoding parameters are decided a priori for all the views.L. Toni and P. Frossard are with Ecole Polytechnique Fédérale de Lausanne (EPFL), Signal Processing Laboratory -LTS4,

Adaptive Streaming in Interactive Multiview Video Systems

“…We now study the performance of our proposed solutions in the scenarios presented in Section V.A, and evaluate their performance in terms of VQM distortion metric [22].…”

“…We now consider the "High-interactivity Scenario". The representations are first optimized at the server with dynamic navigation windows following the model in [22]. The storage constraint per video is 18Mbps as well.…”

“…All these works however are under a simplified assumptions on the client control strategy. Our work is rather complementary to [22] since we focus on optimizing the adaptation logic strategy at the client side. In the following, we only provide the advances that are mostly related to our work and we comment on the recent ones in HAS systems, with a deep focus on MV systems.…”

“…where a, b and e are parameters that depend on both the content characteristics and the resolution of the video, as shown in Table III. These are set to fit experimental (1−VQM) data points, where VQM is an objective Video Quality Metric [22]. Note that a visually pleasant video usually has a VQM score below 0.2 and a reduction in VQM of 0.1 is a significant quality improvement as shown in [44].…”

Optimized receiver control in interactive multiview video streaming systems

“…Only a few works in the literature have studied the optimization of HAS systems for MV streaming. The optimization of the representations to store at the server for HAS MV streaming under simplified assumptions on the client control strategy has been studied in [2]. In this work, we rather target the optimization at the client side for HAS MV as [3], [4].…”