Facilitating Human-Mobile Robot Communication via Haptic Feedback and Gesture Teleoperation

Che, Yuhang; Culbertson, Heather; Tang, Chih‐Wei; Aich, Sudipto; Okamura, Allison M.

doi:10.1145/3243503

Cited by 18 publications

(3 citation statements)

References 43 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, it might be challenging to establish the SA required for teleoperation using solely visual cues (6). Che and colleagues designed a mechatronic device that allows bi-directional communication between a human and a robot follower (12). The device has two parts: a haptic interface that gives users feedback from the robot and a command interface for remote control.…”

Section: Related Workmentioning

confidence: 99%

Electrifying Obstacle Avoidance: Enhancing Teleoperation of Robots with EMS-Assisted Obstacle Avoidance

Shahu,

Kassem,

Zehetgruber

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

We investigate how the use of haptic feedback through electrical muscle stimulation (EMS) can improve collision-avoidance in a robot teleoperation scenario. Background: Collision-free robot teleoperation requires extensive situation awareness by the operator. This is difficult to achieve purely visually when obstacles can exist outside of the robot’s field of view. Therefore, feedback from other sensory channels can be beneficial. Method: We compare feedback modalities in the form of auditory, haptic and bi-modal feedback, notifying users about incoming obstacles outside their field of view, and moving their arms in the direction to avoid the obstacle. We evaluate the different feedback modalities alongside a unimodal visual feedback baseline in a user study (N=9), where participants are controlling a robotic arm in a virtual reality environment. We measure objective performance metrics in terms of the number of collisions and errors, as well as subjective user feedback using the NASA-TLX and the short version of the User Experience Questionnaire. Findings: Unimodal EMS and bi-modal feedback outperformed the baseline and unimodal auditory feedback when it comes to hedonic user experience (p<.001). EMS outperformed the baseline with regards to pragmatic user experience (p=.018). We did not detect significant differences in the performance metrics (collisions and errors). We measured a strong learning effect when investigating the collision count and time. Key insights: The use of EMS is promising for this task. Two of the nine participants reported to experience some level of discomfort. The modality is best utilized for nudging rather than extended movement.

show abstract

Section: Related Workmentioning

confidence: 99%

Electrifying Obstacle Avoidance: Enhancing Teleoperation of Robots with EMS-Assisted Obstacle Avoidance

Shahu,

Kassem,

Zehetgruber

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…To mimic human's capability of cross-modal perception, there have been recent works focusing on the integration of and the conversion between the visual and the tactile data, such as generating the spectrograms of the vibration signals from the visual surface images [17,18], and generating the GelSight-based image according to the visual image of an object surface and vice versa [19,20]. Such visual-tactile data generation could be potentially applied to provide haptic feedback and enable the gestural interface in human-robot interaction and remote communication [21,22].…”

Section: Introductionmentioning

confidence: 99%

Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses

Cai,

Zhu,

Ban

et al. 2021

Preprint

View full text Add to dashboard Cite

Existing psychophysical studies have revealed that the cross-modal visual-tactile perception is common for humans performing daily activities. However, it is still challenging to build the algorithmic mapping from one modality space to another, namely the cross-modal visual-tactile data translation/generation, which could be potentially important for robotic operation. In this paper, we propose a deep-learning-based approach for cross-modal visualtactile data generation by leveraging the framework of the generative adversarial networks (GANs). Our approach takes the visual image of a material surface as the visual data, and the accelerometer signal induced by the pen-sliding movement on the surface as the tactile data. We adopt the conditional-GAN (cGAN) structure together with the residue-fusion (RF) module, and train the model with the additional feature-matching (FM) and perceptual losses to achieve the cross-modal data generation. The experimental results show that the inclusion of the RF module, and the FM and the perceptual losses significantly improves cross-modal data generation performance in terms of the classification accuracy upon the generated data and the visual similarity between the ground-truth and the generated data.

show abstract

“…Along with the advances in virtual-reality and augmented-reality technologies [1,2,3], the user impression of virtual tactile interactions will significantly impact smartphones and smart wearable devices [4,5]. As for the high quality of virtual tactile experiences and long-term working stability of eccentric rotor motors [6,7,8], the quality of the armature is the key factor. Compared with the defect detection of motor armatures of finished products [9], the production line detection based on machine vision is more economical.…”

Section: Introductionmentioning

confidence: 99%

An Industrial Micro-Defect Diagnosis System via Intelligent Segmentation Region

Xia

Wang

Feng

2019

Sensors

View full text Add to dashboard Cite

In the field of machine vision defect detection for a micro workpiece, it is very important to make the neural network realize the integrity of the mask in analyte segmentation regions. In the process of the recognition of small workpieces, fatal defects are always contained in borderline areas that are difficult to demarcate. The non-maximum suppression (NMS) of intersection over union (IOU) will lose crucial texture information especially in the clutter and occlusion detection areas. In this paper, simple linear iterative clustering (SLIC) is used to augment the mask as well as calibrate the score of the mask. We propose an SLIC head of object instance segmentation in proposal regions (Mask R-CNN) containing a network block to learn the quality of the predict masks. It is found that parallel K-means in the limited region mechanism in the SLIC head improved the confidence of the mask score, in the context of our workpiece. A continuous fine-tune mechanism was utilized to continuously improve the model robustness in a large-scale production line. We established a detection system, which included an optical fiber locator, telecentric lens system, matrix stereoscopic light, a rotating platform, and a neural network with an SLIC head. The accuracy of defect detection is effectively improved for micro workpieces with clutter and borderline areas.

show abstract

Facilitating Human-Mobile Robot Communication via Haptic Feedback and Gesture Teleoperation

Cited by 18 publications

References 43 publications

Electrifying Obstacle Avoidance: Enhancing Teleoperation of Robots with EMS-Assisted Obstacle Avoidance

Electrifying Obstacle Avoidance: Enhancing Teleoperation of Robots with EMS-Assisted Obstacle Avoidance

Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses

An Industrial Micro-Defect Diagnosis System via Intelligent Segmentation Region

Contact Info

Product

Resources

About