Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

Abstract

Remote operation of diggers, scalers, and other tunnel-boring machines has significant benefits for worker safety in underground mining. Real-time augmentation of the presented remote views can further improve the operator effectiveness through a more complete presentation of relevant sections of the remote location. In safety-critical applications, such augmentation cannot depend on preconditioned data, nor generate plausible-looking yet inaccurate sections of the view. In this paper, we present a capture and rendering pipeline for real time view augmentation and novel view synthesis that depends only on the inbound data from lidar and camera sensors. We suggest an on-the-fly lidar filtering for reducing point oscillation at no performance cost, and a full rendering process based on lidar depth upscaling and in-view occluder removal from the presented scene. Performance assessments show that the proposed solution is feasible for real-time applications, where per-frame processing fits within the constraints set by the inbound sensor data and within framerate tolerances for enabling effective remote operation.

Augmented remote operating system for scaling in smart mining applications: Quality of experience aspects

“…A less explored area in AT is view augmentation through selective content removal from the presented view. The AROSS [11] is an AT system with augmentation through selective content removal and Novel Perspective view generation. The interplay and the relative benefits of these factors towards operator QoE, especially for the mining context, have not been investigated previously.…”

“…The total time was estimated at approximately 75 minutes per participant. [11]. During Exp 1, an autostereoscopic 4K display was used for presenting the interface of the remote operation system.…”

“…In this study, we explored how in-view augmentations and interface distributions influence the remote operator's ability to perceive wall patterns, and estimate the 3D environment through an Augmented Telepresence (AT) interface. The study was performed with the usage of our Augmented Remote Operating System for Scaling in Mining (AROSS) [11]. The AROSS was initially built to demonstrate the technical feasibility of introducing real-time in-view augmentations (object removal, i.e., "Disocclusion Augmentation") and Novel Perspective view generation for the mining context.…”

“…"NP" denote views showing the scene from positions outside of the camera array at the remote location. Further technical details of AROSS are given in[11]. During Exp 1, an autostereoscopic 4K display was used for presenting the interface of the remote operation system.…”