Learning Forceful Manipulation Skills from Multi-modal Human Demonstrations

Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

“…Human emotion recognition for natural HRI [153,157,158] -Voice user interface for in-vehicle interaction [174] -Audio-based motion generation for HRI [195] User-oriented programming of collaborative robots [96] Audio-vision speech recognition for HRI of the industrial robot and HMI [56,73] VLN for autonomous agent interaction with human and environment [167] Haptics Teaching by demonstration task for HRC [196] -Hardness, temperature and roughness feedback for robot hand controlling and VR applications (haptic glove) [118] Ion-electronic skin providing realtime force directions and strain profiles for various tactile motions (shear, pinch, spread, torsion…) [101] Slip detection and re-grasp planning of unknown objects for robot robust grasping [88] A wearable glove for hand pose and sensory inputs identification for HRI [117] Multimodal robotic sensing system (M-Bot) for HMI [105] PBVS with collision avoidance for safe pHRC [170] Robot self-learning for complex manipulations skills (play Jenga) [89] Robotic manipulators learning from demonstrations for industrial processes such as assembly [108] In-hand pose estimation for robotic assembly [110] Physiology Human activity recognition (MR glass) for hands-free HRI [197] Hand gesture recognition for HMI [97] Wearable glove (hand pose reconstruction, identify sensory inputs such as holding force, object temperature, conductibility, material stiffness, user hear rate) [117] Gesture-based control (EMG þ EEG) to detect and correct robot mistakes for HRI target selection tasks [198] Human motion intention recognition for HRC sawing task [172] Human emotion recognition for HRI and HCI [125,152,157] Lower limb movement prediction for HRI [199] 6.1.…”

“…Such sensors are fundamentally critical for robot action intelligence since robot control in virtually all HRC tasks depends on at least F/T sensor feedback. [ 67,88,89,108–110 ]…”

“…Besides robot learning through self‐experimentation, learning from human demonstration is also a popular approach within HRI. Le et al [ 108 ] employed robot F/T feedback in concert with visual data to replicate learned skills derived from end‐effector poses and measured force. They devised a learned model proficient in merging pose and force features, optimizing stiffness for varying stages of skill acquisition, and implementing an online execution algorithm for adaptive enactment.…”

“…Such sensors are fundamentally critical for robot action intelligence since robot control in virtually all HRC tasks depends on at least F/T sensor feedback. [67,88,89,[108][109][110] Standard pressure sensors like the SynTouch BioTac sensor (https://syntouchinc.com/robotics/), on the other hand, primarily focus on the detection of force values and are habitually installed on robot hand fingers, grippers, and other end effectors. Nonetheless, some advanced tactile sensors [64,65,88,111] are capable of aiding robots in executing more agile and meticulous operations, including in-hand object recognition, detection of subtle collisions, object weighing, slip detection, and touch modality recognition.…”

Interactive Human-in-the-loop Coordination of Manipulation Skills Learned from Demonstration

“…Various teaching methods can be used such as kinesthetic teaching in [5], tele-operation in [8], and visual demonstration in [7]. Different skill models are proposed to abstract these demonstrations: full trajectory of robot end-effector in [6], dynamic movement primitives (DMPs) in [9], [10], task-parameterized Gaussian mixture models (TP-GMMs) in [5], [11] which extend GMMs by incorporating observations from different perspectives (so called task parameters), task-parametrized hidden semi-Markov models (TP-HSMMs) in [12], [13], [14], and deep neural networks [7]. In this work, we adopt the TP-HSMM representation to extract both temporal and spatial features from few human teachings, while allowing generalization over multiple task parameters.…”