NeuralPassthrough: Learned Real-Time View Synthesis for VR

Abstract

Figure 1: We demonstrate the first neural view synthesis method that is optimized to meet the unique requirements for VR passthrough, synthesizing perspective-correct viewpoints in real time and with high visual fidelity. Left: We demonstrate performance using a custom-built VR headset, containing a stereo RGB camera rig with an adjustable baseline. Right: Our method runs in real-time and supports dynamic scenes (top) and near-field objects (bottom).

Geometric distortion on video see‐through head‐mounted displays

“…In addition, shrinking the form factor of VST HMDs is helpful to reduce the camera misalignment in the depth direction. Finally, this study may motivate developing advanced 3D view synthesis techniques 6,19 and distortion correction algorithms that can address the geometric distortion and disocclusion at the same time.…”

“…Unfortunately, it will introduce disocclusion with missing image data at areas close to the edge of the near object (i.e., the occluder), as the position of the VST camera is inconsistent with the eye position. 6…”

“…However, in practice, this requirement is generally violated due to the bulky HMD hardware and challenge to accommodate various interpupillary distance (IPD) among users. Misalignment between the VST camera and user's eye position (both in‐plane and in‐depth) can result in substantial geometric distortion of the physical world, among other image artifacts such as parallax and disocclusion on VST HMDs 4,6 . Geometric distortion, in combination with latency 7 and lack of depth cue, 8 can contribute to visual discomfort and cybersickness 4 .…”

“…Misalignment between the VST camera and user's eye position (both in-plane and in-depth) can result in substantial geometric distortion of the physical world, among other image artifacts such as parallax and disocclusion on VST HMDs. 4,6 Geometric distortion, in combination with latency 7 and lack of depth cue, 8 can contribute to visual discomfort and cybersickness. 4 In addition to degrading the visual experience, geometric distortion of the physical space may potentially raise safety and effectiveness concerns, when using VST HMDs in medical applications, such as surgical procedures.…”

Challenges of Using Augmented Reality to Support an Efficient and Error-Free Assembly in Complex Variant Environments

“…Although newer VR devices may offer the option of what is called a pass-through mode (e.g. Metaquest Pro Headset from Meta), where content is added to a camera capture of the real world (Xiao et al, 2022), VR was not considered as part of this research since there are not many devices available that support this feature and because VR is mostly used in product development and not in productive assembly. (Milgram and Kishino, 1994) AR can be further divided into 3 types: hand-held, head-worn, and spatial AR (van Krevelen and Poelman, 2010).…”