Virtual reality with motion parallax by dense optical flow-based depth generation from two spherical images

Pathak, Sarthak; Moro, Alessandro; Fujii, Hiromitsu; Yamashita, Atsushi; Asama, Hajime

doi:10.1109/sii.2017.8279335

Cited by 7 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classic mathematical approaches exploit camera motion, spline approximations and epipolar geometry to infer relative depth maps [19]- [21]. The work of Pathak et al (2017) [22] takes advantage of the motion parallax phenomena in a virtual reality context. Two spherical images are used to estimate a dense optical flow, which is then decomposed into a relative depth map.…”

Section: B Estimating Depth From Motionmentioning

confidence: 99%

Exploiting Motion Perception in Depth Estimation Through a Lightweight Convolutional Neural Network

Leite

Pinto

2021

IEEE Access

View full text Add to dashboard Cite

Understanding the surrounding 3D scene is of the utmost importance for many robotic applications. The rapid evolution of machine learning techniques has enabled impressive results when depth is extracted from a single image. High-latency networks are required to achieve these performances, rendering them unusable for time-constrained applications. This article introduces a lightweight Convolutional Neural Network (CNN) for depth estimation, NEON, designed for balancing both accuracy and inference times. Instead of solely focusing on visual features, the proposed methodology exploits the Motion-Parallax effect to combine the apparent motion of pixels with texture. This research demonstrates that motion perception provides crucial insight about the magnitude of movement for each pixel, which also encodes cues about depth since large displacements usually occur when objects are closer to the imaging sensor. NEON's performance is compared to relevant networks in terms of Root Mean Squared Error (RMSE), the percentage of correctly predicted pixels (δ 1 ) and inference times, using the KITTI dataset. Experiments prove that NEON is significantly more efficient than the current top ranked network, estimating predictions 12 times faster; while achieving an average RMSE of 3.118 m and a δ 1 of 94.5%. Ablation studies demonstrate the relevance of tailoring the network to use motion perception principles in estimating depth from image sequences, considering that the effectiveness and quality of the estimated depth map is similar to more computational demanding state-of-the-art networks. Therefore, this research proposes a network that can be integrated in robotic applications, where computational resources and processing-times are important constraints, enabling tasks such as obstacle avoidance, object recognition and robotic grasping.

show abstract

Section: B Estimating Depth From Motionmentioning

confidence: 99%

Exploiting Motion Perception in Depth Estimation Through a Lightweight Convolutional Neural Network

Leite

Pinto

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…One benefit is that the sense of "presence" seems to increase in virtual environments in head-mounted displays compared with 2D monitor displays (Kim, Rosenthal, Zielinski, & Brady, 2014). Another benefit is the possibility of implementing a clear sense of depth (Wann, Rushton, & Mon-Williams, 1995) through stereopsis and the use of depth cues such as motion parallax that cannot be extracted from a two-dimensional plane (Pathak, Moro, Fujii, Yamashita, & Asama, 2017). Finally, the affordances of locomoting for successful search completion, change how search is guided in naturalistic settings (Draschkow & Võ, 2016;Hayhoe et al, 2003;Li et al, 2016;Võ, Boettcher & Draschkow, 2019).…”

Section: Using Virtual Realitymentioning

confidence: 99%

Moving foraging into 3D: Feature versus conjunction-based foraging in virtual reality

Kristjánsson¹,

Draschkow²,

Kristjánsson³

et al. 2020

Preprint

View full text Add to dashboard Cite

Visual attention evolved in a three-dimensional world, yet studies on human attention in three- dimensions are sparse. Here we present findings from a human foraging study in immersive three-dimensional virtual reality. We used a foraging task introduced in Kristjánsson et al. (2014) to examine how well their findings generalize to more naturalistic settings. A second goal was to examine what effect the motion of targets and distractors has on intertarget times, run patterns and foraging organization. Observers foraged for 50 targets among 50 distractors in four different conditions. Targets were distinguished from distractors by either a single feature (feature foraging) or a conjunction of features (conjunction foraging). Further, those conditions were performed both with static and moving targets and distractors. Our results replicate previous foraging studies in many aspects, with constant intertarget times during a “cruise-phase” within foraging trials and response time peaks at the end of foraging trials. Some key differences emerged, however, such as more frequent switches between target types during conjunction foraging than previously seen and a lack of clear mid-peaks during conjunction foraging, possibly reflecting that differences between feature and conjunction processing are smaller within 3D environments. Observers initiated their foraging in the bottom part of the visual field and motion did not have much of an effect on selection times between different targets (intertarget times, ITT’s) or run behaviour patterns except for the end-peaks. Our results cast new light upon visual attention in three-dimensional environments and highlight how 3D virtual reality studies can provide important extensions to two-dimensional studies of visual attention.

show abstract

“…There are only few works [21]- [23] that estimate depth from pairs of views. The studies [21], [22] (and their prior analyses) estimate the relative pose between the cameras using traditional A-KAZE features [24] and the eight-point algorithm (8-PA) [25]; derotate and estimate dense features from the views; and refine the pose using an iterative nonlinear approach. Lai and colleagues [23] present an encoderdecoder model that deals with stereo-rectfied image pairs with a small, fixed baseline.…”

Section: Related Workmentioning

confidence: 99%

Dense 3D Indoor Scene Reconstruction from Spherical Images

Silveira¹,

Jung

2020

Anais Estendidos Da Conference on Graphics, Patterns and Images (SIBRAPI Estendido 2020)

View full text Add to dashboard Cite

Techniques for 3D reconstruction of scenes based on images are popular and support a number of secondary applications. Traditional approaches require several captures for covering whole environments due to the narrow field of view (FoV) of the pinhole-based/perspective cameras. This paper summarizes the main contributions of the homonym Ph.D. Thesis, which addresses the 3D scene reconstruction problem by considering omnidirectional (spherical or 360◦ ) cameras that present a 360◦ × 180◦ FoV. Although spherical imagery have the benefit of the full-FoV, they are also challenging due to the inherent distortions involved in the capture and representation of such images, which might compromise the use of many wellestablished algorithms for image processing and computer vision. The referred Ph.D. Thesis introduces novel methodologies for estimating dense depth maps from two or more uncalibrated and temporally unordered 360◦ images. It also presents a framework for inferring depth from a single spherical image. We validate our approaches using both synthetic data and computer-generated imagery, showing competitive results concerning other state-ofthe-art methods.

show abstract

Virtual reality with motion parallax by dense optical flow-based depth generation from two spherical images

Cited by 7 publications

References 10 publications

Exploiting Motion Perception in Depth Estimation Through a Lightweight Convolutional Neural Network

Exploiting Motion Perception in Depth Estimation Through a Lightweight Convolutional Neural Network

Moving foraging into 3D: Feature versus conjunction-based foraging in virtual reality

Dense 3D Indoor Scene Reconstruction from Spherical Images

Contact Info

Product

Resources

About