Point Cloud-Based Model-Mediated Teleoperation With Dynamic and Perception-Based Model Updating

Abstract

In this paper, we extend the concept of model-mediated teleoperation (MMT) for complex environments and six degrees of freedom interaction using point cloud surface models. In our system, a time-of-flight camera is used to capture a high resolution point cloud model of the object surface. The point cloud model and the physical properties of the object (stiffness and surface friction coefficient) are estimated at the slave side in real-time and transmitted to the master side using the modeling and updating algorithm proposed in this work. The proposed algorithm adaptively controls the updating of the point cloud model and the object properties according to the slave movements and by exploiting known limitations of human haptic perception. As a result, perceptually irrelevant transmissions are avoided, and thus the packet rate in the communication channel is substantially reduced. In addition, a simple point cloud-based haptic rendering algorithm is adopted to generate the force feedback signals directly from the point cloud model without first converting it into a 3D mesh. In the experimental evaluation, the system stability and transparency are verified in the presence of a round-trip communication delay of up to 1000ms. Furthermore, by exploiting the limits of human haptic perception the presented system allows for a significant haptic data reduction of about 90% for teleoperation systems with time delay. Index Termsmodel-mediated teleoperation, model-update, packet rate reduction, point cloud-based haptic rendering.

Toward Haptic Communications Over the 5G Tactile Internet

“…Extrapolation is used when the slave device needs to operate outside the area produced by the point-cloud model, although issues are very likely to emerge. The authors of [200] also reported a data reduction of about 90% with guaranteed (significantly high) subjective quality of teleoperation. Augmented feedback information was also considered in [201] again with the use of a stereo camera.…”

“…This combination of perceptual and statistical methods has been made in [199] by first applying the JND threshold and then a double exponential smoothing prediction algorithm to fill in the values not transmitted due to the threshold. Xu et al [200] also applied the perceptual deadband approach to the estimated model parameters to reduce the transmission rate. The proposed framework incorporates 3D sensors to produce a point cloud model of a static rigid object's surface in the remote environment.…”

Model Mediation to Overcome Light Limitations—Toward a Secure Tactile Internet System

“…In [75], authors proposed a point-cloud-based model mediated (Pcbmm) system for bilateral communication in a transparent and stable way. The system is developed for complex remote environments and these hard environments are modeled using point clouds rather simple geometry approximation.…”

Toward QoE-driven dynamic control scheme switching for time-delayed teleoperation systems: A dedicated case Study

“…State-of-the-art solutions that address the aforementioned teleoperation challenges (sensitivity to delay and high packet rate) focus on combining different stability-ensuring control schemes with haptic packet rate reduction methods such as [5], [6].…”