Exploring gestural input for engineering surveys of real-life structures in virtual reality using photogrammetric 3D models

Abstract

Photogrammetry is a promising set of methods for generating photorealistic 3D models of physical objects and structures. Such methods may rely solely on camera-captured photographs or include additional sensor data. Digital twins are digital replicas of physical objects and structures. Photogrammetry is an opportune approach for generating 3D models for the purpose of preparing digital twins. At a sufficiently high level of quality, digital twins provide effective archival representations of physical objects and structures and become effective substitutes for engineering inspections and surveying. While photogrammetric techniques are well-established, insights about effective methods for interacting with such models in virtual reality remain underexplored. We report the results of a qualitative engineering case study in which we asked six domain experts to carry out engineering measurement tasks in an immersive environment using bimanual gestural input coupled with gaze-tracking. The qualitative case study revealed that gaze-supported bimanual interaction of photogrammetric 3D models is a promising modality for domain experts. It allows the experts to efficiently manipulate and measure elements of the 3D model. To better allow designers to support this modality, we report design implications distilled from the feedback from the domain experts.

Measurement and Inspection of Photo-Realistic 3-D VR Models

“…The data analysis suggests a range of research avenues for future work, such as refining the hand-menu layout and the toolkit design. Another fruitful area for future investigations is to investigate the effectiveness of measurements of other real-world quantities of a structure, such as its size, surface area, or volume [5].…”

“…The hardware setup consisted of the Oculus Rift HMD, its position-tracking sensor, and the Leap Motion hand-tracking sensor. The basis of the VR framework underpinning the system was previously developed for aeroengine digital twin assessment [4] and used in another qualitative user study [5]. This prior user study [5] was concerned with a different photogrammetric model and the participants in that study were chosen due to their expertise in civil engineering rather than piping systems.…”

“…The basis of the VR framework underpinning the system was previously developed for aeroengine digital twin assessment [4] and used in another qualitative user study [5]. This prior user study [5] was concerned with a different photogrammetric model and the participants in that study were chosen due to their expertise in civil engineering rather than piping systems. The VR environment is demonstrated by being populated with a photo-realistic reconstruction of an existing pipeline system (see Fig.…”

“…We used a mixed-methods approach with three domain experts and two different models [5] to assess the suitability of the system. After watching a short video clip, the participants were asked to perform slightly different tasks depending on which model they were working with.…”

Guest editorial: Interaction in immersive experiences

“…The next three papers in this special issue focus on emerging input and interaction approaches appropriate for immersive experiences. Tadeja et al [14] propose an approach for bimanual gestural interaction in VR coupled with gaze-tracking. The feasibility of the approach is tested in an engineering case study to support the manipulation and measurement of complex photogrammetric 3D structure models.…”