A multimodal user interface for touchless control of robotic ultrasound

Abstract

Purpose Past research contained the investigation and development of robotic ultrasound. In this context, interfaces which allow for interaction with the robotic system are of paramount importance. Few researchers have addressed the issue of developing non-tactile interaction approaches, although they could be beneficial for maintaining sterility during medical procedures. Interaction could be supported by multimodality, which has the potential to enable intuitive and natural interaction. To assess the feasibility of multimodal interaction for non-tactile control of a co-located robotic ultrasound system, a novel human–robot interaction concept was developed. Methods The medical use case of needle-based interventions under hybrid computed tomography and ultrasound imaging was analyzed by interviewing four radiologists. From the resulting workflow, interaction tasks were derived which include human–robot interaction. Based on this, characteristics of a multimodal, touchless human–robot interface were elaborated, suitable interaction modalities were identified, and a corresponding interface was developed, which was thereafter evaluated in a user study with eight participants. Results The implemented interface includes voice commands, combined with hand gesture control for discrete control and navigation interaction of the robotic US probe, respectively. The interaction concept was evaluated by the users in the form of a quantitative questionnaire with a average usability. Qualitative analysis of interview results revealed user satisfaction with the implemented interaction methods and potential improvements to the system. Conclusion A multimodal, touchless interaction concept for a robotic US for the use case of needle-based procedures in interventional radiology was developed, incorporating combined voice and hand gesture control. Future steps will include the integration of a solution for the missing haptic feedback and the evaluation of its clinical suitability.

Gesture Recognition in Virtual reality

“…Smartphones often incorporate gesture recognition for tasks like navigating through photos or answering calls. In healthcare, gesture-controlled interfaces can be used for touchless control of medical equipment (Josefine et al, 2022), enhancing hygiene in clinical settings. Gesture recognition is also prevalent in augmented reality (AR) and virtual reality (VR) systems, allowing users to interact with digital environments through natural movements.…”

Development of a Touchless Control System for a Clinical Robot with Multimodal User Interface

“…Multimodal user interfaces empower users to seamlessly transition between diverse control modes contingent upon their proclivities or the specific task at hand [6]. As an illustration, a user might seamlessly oscillate between voice commands to oversee the robot's locomotion, while seamlessly transitioning to gesture-based control for tasks demanding precision in manipulation [7].…”