Embodied Robot Teleoperation Based on High-Fidelity Visual-Haptic Simulator: Pipe-Fitting Example

Zhou, Tianyu; Xia, Pengxiang; Ye, Yang; Du, Jing

doi:10.1061/jcemd4.coeng-13916

Cited by 9 publications

(4 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We utilized ROS and Unity game engine to facilitate the RC functions, with Unity specializing in real-time modeling of human motion data and providing a robust platform for developing immersive and intuitive VR working spaces for operators. Haptic device control data from the human operator is captured and processed in Unity before being streamed to ROS for real-time robot control, based on our previous works 1 , 3 . The Emika Panda robot is used to as the robotic model in the proposed architecture.…”

Section: Methodsmentioning

confidence: 99%

“…Robot teleoperation, the technique of controlling robots from a distance, facilitates human interaction with environments that are remote, hazardous, or inaccessible 1 . Through teleoperation, operators can engage in intricate tasks, surpassing the limitations imposed by the spatial separation between human and robot, and thereby, expanding the horizons of applications such as deep-sea exploration, space missions, and hazardous material handling 2 – 5 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

Du,

Vann,

Zhou

et al. 2024

Sci Rep

Self Cite

View full text Add to dashboard Cite

In the realm of robotics and automation, robot teleoperation, which facilitates human–machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

Du,

Vann,

Zhou

et al. 2024

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…Robot teleoperation enables human operators to command and control robots in distant, hazardous, or inaccessible environments (Senft et al, 2021 ). This ability expands the range of feasible applications, such as deep-sea exploration, space missions, and hazardous material handling, allowing for complex tasks to be conducted beyond the conventional spatial limitations imposed between the human operator and the robot (Zhou et al, 2023 ). However, the potential of teleoperation is often undermined by operational delays due to physical constraints like signal transmission distances and processing limits, resulting in latency that affects situational awareness, control precision, and task performance (Kluge et al, 2013 ; Wenhao et al, 2017 ; Payra et al, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

Neural dynamics of delayed feedback in robot teleoperation: insights from fNIRS analysis

Zhou,

Ye,

Zhu

et al. 2024

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

IntroductionAs robot teleoperation increasingly becomes integral in executing tasks in distant, hazardous, or inaccessible environments, operational delays remain a significant obstacle. These delays, inherent in signal transmission and processing, adversely affect operator performance, particularly in tasks requiring precision and timeliness. While current research has made strides in mitigating these delays through advanced control strategies and training methods, a crucial gap persists in understanding the neurofunctional impacts of these delays and the efficacy of countermeasures from a cognitive perspective.MethodsThis study addresses the gap by leveraging functional Near-Infrared Spectroscopy (fNIRS) to examine the neurofunctional implications of simulated haptic feedback on cognitive activity and motor coordination under delayed conditions. In a human-subject experiment (N = 41), sensory feedback was manipulated to observe its influences on various brain regions of interest (ROIs) during teleoperation tasks. The fNIRS data provided a detailed assessment of cerebral activity, particularly in ROIs implicated in time perception and the execution of precise movements.ResultsOur results reveal that the anchoring condition, which provided immediate simulated haptic feedback with a delayed visual cue, significantly optimized neural functions related to time perception and motor coordination. This condition also improved motor performance compared to the asynchronous condition, where visual and haptic feedback were misaligned.DiscussionThese findings provide empirical evidence about the neurofunctional basis of the enhanced motor performance with simulated synthetic force feedback in the presence of teleoperation delays. The study highlights the potential for immediate haptic feedback to mitigate the adverse effects of operational delays, thereby improving the efficacy of teleoperation in critical applications.

show abstract

“…Notably, in the context of industrial robot control interfaces, there exists a noticeable gap that limits human operators' perception of the situation [21]. Current solutions, relying on visual feedback like camera feeds, pose challenges in spatial comprehension, including the assessment of distances between robots and obstacles [22].…”

Section: Introductionmentioning

confidence: 99%

Virtual Reality Teleoperation System for Mobile Robot Manipulation

Galarza,

Ayala,

Manzano

et al. 2023

Robotics

View full text Add to dashboard Cite

Over the past few years, the industry has experienced significant growth, leading to what is now known as Industry 4.0. This advancement has been characterized by the automation of robots. Industries have embraced mobile robots to enhance efficiency in specific manufacturing tasks, aiming for optimal results and reducing human errors. Moreover, robots can perform tasks in areas inaccessible to humans, such as hard-to-reach zones or hazardous environments. However, the challenge lies in the lack of knowledge about the operation and proper use of the robot. This work presents the development of a teleoperation system using HTC Vive Pro 2 virtual reality goggles. This allows individuals to immerse themselves in a fully virtual environment to become familiar with the operation and control of the KUKA youBot robot. The virtual reality experience is created in Unity, and through this, robot movements are executed, followed by a connection to ROS (Robot Operating System). To prevent potential damage to the real robot, a simulation is conducted in Gazebo, facilitating the understanding of the robot’s operation.

show abstract

Embodied Robot Teleoperation Based on High-Fidelity Visual-Haptic Simulator: Pipe-Fitting Example

Cited by 9 publications

References 84 publications

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

Neural dynamics of delayed feedback in robot teleoperation: insights from fNIRS analysis

Virtual Reality Teleoperation System for Mobile Robot Manipulation

Contact Info

Product

Resources

About