Proxy‐guided Image‐based Rendering for Mobile Devices

Abstract

Mesh warp 2.3 fps Our method 35.4 fps Ground truthFigure 1: Our new image-based rendering algorithm warps original views (with depth, not shown) to produce novel views. It performs as fast as the fastest competitors but maintains much higher visual quality for larger displacements. The insets compare against other common approaches and show the RGB-DSSIM difference. Frame rates are computed on an an Intel Compute Stick at a resolution of 1280×720.Abstract VR headsets and hand-held devices are not powerful enough to render complex scenes in real-time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image-based rendering (IBR) architecture for masking the latency. It runs in real-time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server-side dual-view representation that leverages an optimally-placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide-angle projection with favorable directional parameterization. A new client-side IBR algorithm uses a pre-transmitted level-of-detail proxy with an encaging simplification and depth-carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR / mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.

Tessellated Shading Streaming

“…In many video games the background geometry remains static. This fact can be exploited as the static geometry can be used directly for perspective texture mapping, optionally with further simplification [RKR∗16]. Keeping the full model data on client and server allows to reduce transmission only to images after factoring in low‐frequency illumination [Lev95], disocclusions [MC097] or removing complex indirect illumination effects [CLM∗15].…”

Compressed Animated Light Fields with Real-Time View-Dependent Reconstruction

“…Outatime generates new frames by employing speculative execution, in order to mitigate wide-area latency [18]. Another approach is to use a novel wide-angle projection and dual-view pairs to synthesize a new image: a primary view and a secondary view at quarter resolution to approximately resolve disocclusions [25]. Szirmay-Kalos et al [27] use color and depth cubemaps to approximate ray-traced effects.…”

CloudVR

“…It is unsuitable for interactive VR applications that require real-time rendering. Reinert et al proposed an image based rendering (IBR) system for VR [21]. Relying on IBR, it is also designed for static scenes although the authors also propose some fallbacks to handle dynamic scenes in the paper.…”