A virtual surgical prototype system based on gesture recognition for virtual surgical training in maxillofacial surgery

Abstract

Background Virtual reality (VR) technology is an ideal alternative of operation training and surgical teaching. However, virtual surgery is usually carried out using the mouse or data gloves, which affects the authenticity of virtual operation. A virtual surgery system with gesture recognition and real-time image feedback was explored to realize more authentic immersion. Method Gesture recognition technology proposed with an efficient and real-time algorithm and high fidelity was explored. The recognition of hand contour, palm and fingertip was firstly realized by hand data extraction. Then, an Support Vector Machine classifier was utilized to classify and recognize common gestures after extraction of feature recognition. The algorithm of collision detection adopted Axis Aligned Bounding Box binary tree to build hand and scalpel collision models. What’s more, nominal radius theorem (NRT) and separating axis theorem (SAT) were applied for speeding up collision detection. Based on the maxillofacial virtual surgical system we proposed before, the feasibility of integration of the above technologies in this prototype system was evaluated. Results Ten kinds of signal static gestures were designed to test gesture recognition algorithms. The accuracy of gestures recognition is more than 80%, some of which were over 90%. The generation speed of collision detection model met the software requirements with the method of NRT and SAT. The response time of gesture] recognition was less than 40 ms, namely the speed of hand gesture recognition system was greater than 25 Hz. On the condition of integration of hand gesture recognition, typical virtual surgical procedures including grabbing a scalpel, puncture site selection, virtual puncture operation and incision were carried out with realization of real-time image feedback. Conclusion Based on the previous maxillofacial virtual surgical system that consisted of VR, triangular mesh collision detection and maxillofacial biomechanical model construction, the integration of hand gesture recognition was a feasible method to improve the interactivity and immersion of virtual surgical operation training.

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

“…This study exhibits a certain degree of innovation in the field of VR interaction, particularly in exploring the combined use of gestures and voice for operational purposes. Current literature primarily focuses on gesture-based operations [5,6,7,8,9]. These studies provide a crucial background for understanding the application of gestures in VR, yet the exploration of combining gestures with voice interaction remains relatively limited.…”

“…However, the prevalent reliance on traditional input devices like keyboards, mice, or even touchscreens in VR environments limits the naturalness and intuitiveness of user interactions [3,4]. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition [5,6,7,8,9]. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments [10].Recognizing the need for more immersive and user-friendly interfaces, this research focuses on the development of a VR interaction method based on gesture and voice recognition.…”

An End-to-End Model of ArVi-MoCoGAN and C3D with Attention Unit for Arbitrary-view Dynamic Gesture Recognition

“…Human gesture recognition is an attractive field in computer vision with many applications such as human computer interaction, human behavior analysis, intelligent surveillance, and virtual reality [1], [2]. A recognition system could use (1) static gestures and (2) dynamic gestures.…”