Stereoscopic First Person View System for Drone Navigation

Abstract

Ground control of unmanned aerial vehicles (UAV) is a key to the advancement of this technology for commercial purposes. The need for reliable ground control arises in scenarios where human intervention is necessary, e.g. handover situations when autonomous systems fail. Manual flights are also needed for collecting diverse datasets to train deep neural network-based control systems. This axiom is even more prominent for the case of unmanned flying robots where there is no simple solution to capture optimal navigation footage. In such scenarios, improving the ground control and developing better autonomous systems are two sides of the same coin. To improve the ground control experience, and thus the quality of the footage, we propose to upgrade onboard teleoperation systems to a fully immersive setup that provides operators with a stereoscopic first person view (FPV) through a virtual reality (VR) head-mounted display. We tested users (n = 7) by asking them to fly our drone on the field. Test flights showed that operators flying our system can take off, fly, and land successfully while wearing VR headsets. In addition, we ran two experiments with prerecorded videos of the flights and walks to a wider set of participants (n = 69 and n = 20) to compare the proposed technology to the experience provided by current drone FPV solutions that only include monoscopic vision. Our immersive stereoscopic setup enables higher accuracy depth perception, which has clear implications for achieving better teleoperation and unmanned navigation. Our studies show comprehensive data on the impact of motion and simulator sickness in case of stereoscopic setup. We present the device specifications as well as the measures that improve teleoperation experience and reduce induced simulator sickness. Our approach provides higher perception fidelity during flights, which leads to a more precise better teleoperation and ultimately translates into better flight data for training deep UAV control policies.

Geovisualisation of Relief in a Virtual Reality System on the Basis of Low-Level Aerial Imagery

“…The other application of UAV-based terrain reconstruction within the VR system are simulators. VR FPV mode interfaces can improve the operator situational awareness and provide valuable tools to help understand and analyse the vehicle surroundings and plan command sequences Nguyen et al (2000), Postal et al (2016) and Smolyanskiy and Gonzalez-Franco (2017).…”

The Effects of Visual and Control Latency on Piloting a Quadcopter Using a Head-Mounted Display

“…In general, operating a vehicle with an HMD and a joystick does not provide performance advantages compared with operating a vehicle with a computer monitor and a joystick. The former [10][11][12][13][14] usually elicits symptoms of simulator sickness. However, operating vehicles using only HMDs [1][2][3][4] seem to give people a more direct, immersive and intuitive way of control than joint use of computer monitors and joysticks.…”

“…A more recent work by Smolyanskiy and Gonzalez-Franco [14] presented a design of a quadcopter with stereoscopic viewing using the Oculus Rift DK2. Onboard camera panning was controlled digitally (as opposed to mechanically) by head motion while the quadcopter was piloted by a hand-held controller.…”

Pulse and vital sign measurement in mixed reality using a HoloLens

“…ese emotions can be measured through physiological signals [Meehan et al 2002;Padrao et al 2016;Spanlang et al 2014]. Importantly, monitoring the physiological state of the wearer and those around them in a non-invasive way could help indicate undesirable e ects, such as simulator sickness [Smolyanskiy and Gonzalez-Franco 2017]. For example, the system could recommend participants take a break from the device when they exceed certain physiological thresholds.…”