Control framework for collaborative robot using imitation learning-based teleoperation from human digital twin to robot digital twin

Lee, Hyun Soo; Kim, Seong Dae; Amin, Mohammad Aman Ullah Al

doi:10.1016/j.mechatronics.2022.102833

Cited by 12 publications

(6 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Search algorithms [59] Decision making [59] Assembly line reconfiguration and planning [59] Neural networks FFN [57] RNN [73] Object recognition [57] Sequential data handling [73] Human safety [57] Dynamic changes' prediction [73] Deep learning 1D-CNN [74] Mask R-CNN [75,76] CNN [53,58,60,77,78] PVNet Parallel network [79] PointNet [79] SAE [80] Detection or recognition [53,60,[74][75][76]79] Classification [77] Human action and motion recognition [78] Pose estimation [58,79] Feature extraction [73] Anomaly detection [80] Human safety [60,75,76,[78][79][80] Ergonomics [60] Human action and intention understanding [60,77] Efficiency [78,79] Position estimation [53] Decision making [73] Object manipulation [74] Reinforcement learning Model-free RL…”

Section: Ai Methods Specific Technique Task Application Problemmentioning

confidence: 99%

“…Therefore, the next subsections will discuss enabling technologies and methods in various use cases and analyse how they enable human-centric DT in HMC applications. DT of physical robot [51] DT of virtual space [52] MR system development [53] Human action prediction [54] Safety and productivity [55] Human reaction analysis [56] Matlab [57,58] High-level technical computing language Engineering, Research…”

Section: Digital Twins and Simulationsmentioning

confidence: 99%

“…Obstacle detection and 3D localisation [57] Human digital twin [58] ROS [55,[59][60][61][62] Middleware for robotics software development Robotics, automation…”

Section: Digital Twins and Simulationsmentioning

confidence: 99%

See 2 more Smart Citations

Towards a Human-Centric Digital Twin for Human–Machine Collaboration: A Review on Enabling Technologies and Methods

Krupas,

Kajati,

Liu

et al. 2024

Sensors

View full text Add to dashboard Cite

With the intent to further increase production efficiency while making human the centre of the processes, human-centric manufacturing focuses on concepts such as digital twins and human–machine collaboration. This paper presents enabling technologies and methods to facilitate the creation of human-centric applications powered by digital twins, also from the perspective of Industry 5.0. It analyses and reviews the state of relevant information resources about digital twins for human–machine applications with an emphasis on the human perspective, but also on their collaborated relationship and the possibilities of their applications. Finally, it presents the results of the review and expected future works of research in this area.

show abstract

Section: Ai Methods Specific Technique Task Application Problemmentioning

confidence: 99%

Section: Digital Twins and Simulationsmentioning

confidence: 99%

See 1 more Smart Citation

Towards a Human-Centric Digital Twin for Human–Machine Collaboration: A Review on Enabling Technologies and Methods

Krupas,

Kajati,

Liu

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…In [14], Tao et al used 3D pose baseline to predict the 3D human pose in an RGB image. In [15], the 3D pose estimation was also used to imitate. With this method, we only need to prepare a normal camera or a video to capture 3D human motion.…”

Section: Related Work a Motion Imitation Methods For Humanoid Robotmentioning

confidence: 99%

Imitation System of Humanoid Robots and Its Applications

Zhan¹,

Huang²

2023

IEEE Open J. Circuits Syst.

View full text Add to dashboard Cite

In this paper, we propose an imitation system that imitates human motions in videos to plan robot actions that are similar to human motions, with the aim of the complicated whole-body action planning of humanoid robots. Additionally, we created an interaction system that will enable basic human-robot interaction for our humanoid robot. To obtain the 3D coordinates of the key points on the human body, we used the 3D pose estimation model. The key points were then transformed into various trajectory files needed by the robot to complete the motion, using the mapping method proposed in this research, which refers to the control strategy and stability of the robot. In addition, we proposed some post-processing methods to post-process the trajectories. In the interaction system, we created a speech and vision system so that the robot could detect human gestures or postures and converse with people. It also has a music rhythm recognition system developed by seniors that enables the robot to dance to the beats of the song. Finally, through this system, we completed several human-robot interaction scenarios, which proved the convenience, and effectiveness of motion planning with an imitation system, and the completeness of the interaction system.

show abstract

“…Achieving real-time synchronization between the digital twin and the physical robot is crucial for seamless robotic telemanipulation [129]. Ensuring data accuracy and calibration poses another significant challenge, as digital twins heavily depend on precise sensor data to accurately represent the state of the physical robot [130]. Additionally, creating detailed and accurate digital twin models for complex robotic systems presents a formidable obstacle [131,132].…”

Section: Mixed Reality-based Vision Mapping and Mergingmentioning

confidence: 99%

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

Su,

Chen,

Zhou

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

show abstract

Control framework for collaborative robot using imitation learning-based teleoperation from human digital twin to robot digital twin

Cited by 12 publications

References 35 publications

Towards a Human-Centric Digital Twin for Human–Machine Collaboration: A Review on Enabling Technologies and Methods

Towards a Human-Centric Digital Twin for Human–Machine Collaboration: A Review on Enabling Technologies and Methods

Imitation System of Humanoid Robots and Its Applications

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

Contact Info

Product

Resources

About