Teleoperation Control of an Underactuated Bionic Hand: Comparison between Wearable and Vision-Tracking-Based Methods

Fu, J.; Poletti, Massimiliano; Liu, Qingsheng; Iovene, Elisa; Su, Hang; Ferrigno, Giancarlo; Momi, Elena De

doi:10.3390/robotics11030061

Cited by 10 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the camera frame was estimated. In addition, a Kalman Filter [25] was adopted to minimise the reconstruction noise caused by noisy image coordinates estimation. The motion of the instrument was modeled with a 3D constant velocity model and was used to predict the position of the tool at the next time step 𝑛 + 1:…”

Section: A Vision Modulementioning

confidence: 99%

Towards Exoscope Automation in Neurosurgery: A Markerless Visual-Servoing Approach

Iovene

Casella

Iordache

et al. 2023

IEEE Trans. Med. Robot. Bionics

View full text Add to dashboard Cite

Exoscopes are a promising tool for neurosurgeons, offering improved visualisation and ergonomics compared with traditional surgical microscopes. They consist of an external scope that projects the surgical field onto a 2D or 3D monitor, providing a wider field of view and better access to the surgical site. Despite the advantages, exoscopes present some limitations, such as the need for manual or foot joystick repositioning, which can disrupt the flow of the procedure and increase the risk of user error. In this study, a markerless visual-servoing approach for autonomous exoscope control is proposed to address these limitations and enhance the ergonomics and reduce the physical and cognitive load compared with traditional joystick control. The system uses visual information from the operating field to control the exoscope, eliminating the need for markers or additional tracking devices. The proposed approach was validated using a 7-DOF robotic manipulator with a stereo camera in an eye-in-hand configuration. Results showed that the system achieved 89 % accuracy in detecting the target and tracking its movement with a tracking error ranging from 0.50 ± 0.17 cm for low-speed movements to 1.38 ± 0.73 cm for high-speed movements. The proposed system also demonstrated improved efficiency, with a shorter execution time of 72.07 ± 19.36 s compared with 106.52±18.50 s for the foot-joystick control. Additionally, the time out of the FoV was significantly higher in the joystick control mode and the frequency of appearance of the instrument in the centre of the image was higher when using the proposed system. The NASA TLX results indicated lower physical and cognitive load compared with the joystick control-based modality.

show abstract

Section: A Vision Modulementioning

confidence: 99%

Towards Exoscope Automation in Neurosurgery: A Markerless Visual-Servoing Approach

Iovene

Casella

Iordache

et al. 2023

IEEE Trans. Med. Robot. Bionics

View full text Add to dashboard Cite

show abstract

“…Alternatively, the hand can be modeled as a rigid body in space. The linearized motion model of a rigid body in space can be written as [52] x…”

Section: Multi-sensor Data Fusion and Control Strategy 231 Sensor Fus...mentioning

confidence: 99%

Whole-Body Teleoperation Control of Dual-Arm Robot Using Sensor Fusion

Wang,

Chen,

Dong

et al. 2023

Biomimetics

Self Cite

View full text Add to dashboard Cite

As human–robot interaction and teleoperation technologies advance, anthropomorphic control of humanoid arms has garnered increasing attention. However, accurately translating sensor-detected arm motions to the multi-degree freedom of a humanoid robotic arm is challenging, primarily due to occlusion issues with single-sensor setups, which reduce recognition accuracy. To overcome this problem, we propose a human-like arm control strategy based on multi-sensor fusion. We defined the finger bending angle to represent finger posture and employed a depth camera to capture arm movement. Consequently, we developed an arm movement tracking system and achieved anthropomorphic control of the imitation human arm. Finally, we verified our proposed method’s effectiveness through a series of experiments, evaluating the system’s robustness and real-time performance. The experimental results show that this control strategy can control the motion of the humanoid arm stably, and maintain a high recognition accuracy in the face of complex situations such as occlusion.

show abstract

“…The movement of the bionic bimanual robot requires a high degree of coordination and complex manipulation. Bionic bimanual robot teleoperation [ 5 ] can transfer the operation skills of the human dual hands to the robots to increase the success accuracy and reliability of the manipulation, which is the first choice for smart programming solution when dealing with fast decisions and corner cases.…”

Section: Introductionmentioning

confidence: 99%

Dual-Hand Motion Capture by Using Biological Inspiration for Bionic Bimanual Robot Teleoperation

Gao,

Deng,

et al. 2023

Cyborg Bionic Syst

View full text Add to dashboard Cite

Bionic bimanual robot teleoperation can transfer the grasping and manipulation skills of human dual hands to the bionic bimanual robots to realize natural and flexible manipulation. The motion capture of dual hands plays an important role in the teleoperation. The motion information of dual hands can be captured through the hand detection, localization, and pose estimation and mapped to the bionic bimanual robots to realize the teleoperation. However, although the motion capture technology has achieved great achievements in recent years, visual dual-hand motion capture is still a great challenge. So, this work proposed a dual-hand detection method and a 3-dimensional (3D) hand pose estimation method based on body and hand biological inspiration to achieve convenient and accurate monocular dual-hand motion capture and bionic bimanual robot teleoperation. First, a dual-hand detection method based on body structure constraints is proposed, which uses a parallel structure to combine hand and body relationship features. Second, a 3D hand pose estimation method with bone-constraint loss from single RGB images is proposed. Then, a bionic bimanual robot teleoperation method is designed by using the proposed hand detection and pose estimation methods. Experiment results on public hand datasets show that the performances of the proposed hand detection and 3D hand pose estimation outperform state-of-the-art methods. Experiment results on a bionic bimanual robot teleoperation platform shows the effectiveness of the proposed teleoperation method.

show abstract

Teleoperation Control of an Underactuated Bionic Hand: Comparison between Wearable and Vision-Tracking-Based Methods

Cited by 10 publications

References 34 publications

Towards Exoscope Automation in Neurosurgery: A Markerless Visual-Servoing Approach

Towards Exoscope Automation in Neurosurgery: A Markerless Visual-Servoing Approach

Whole-Body Teleoperation Control of Dual-Arm Robot Using Sensor Fusion

Dual-Hand Motion Capture by Using Biological Inspiration for Bionic Bimanual Robot Teleoperation

Contact Info

Product

Resources

About