A robust method for VR-based hand gesture recognition using density-based CNN

Abstract

Many VR-based medical purposes applications have been developed to help patients with mobility decrease caused by accidents, diseases, or other injuries to do physical treatment efficiently. VR-based applications were considered more effective helper for individual physical treatment because of their low-cost equipment and flexibility in time and space, less assistance of a physical therapist. A challenge in developing a VR-based physical treatment was understanding the body part movement accurately and quickly. We proposed a robust pipeline to understanding hand motion accurately. We retrieved our data from movement sensors such as HTC vive and leap motion. Given a sequence position of palm, we represent our data as binary 2D images of gesture shape. Our dataset consisted of 14 kinds of hand gestures recommended by a physiotherapist. Given 33 3D points that were mapped into binary images as input, we trained our proposed density-based CNN. Our CNN model concerned with our input characteristics, having many 'blank block pixels', 'single-pixel thickness' shape and generated as a binary image. Pyramid kernel size applied on the feature extraction part and classification layer using softmax as loss function, have given 97.7% accuracy.

A Novel Sensor Fusion Approach for Precise Hand Tracking in Virtual Reality-Based Human—Computer Interaction

“…Gesture recognition, a fundamental aspect of hand tracking, allows users to communicate with virtual environments through a variety of hand gestures. In the study by Liliana et al [ 42 ], a pipeline combining movement sensors, a binary image representation of a gesture shape, and a density-based CNN was proposed. This pipeline, combined with HTC Vive, Kinect, and Leap Motion, has achieved accurate hand gesture recognition with a remarkable 97.7% accuracy.…”

Virtual reality's effects on air crash accident investigation learning interaction

“…Another example of the application of visualisation may be seen in the interpretation of sign language through the use of hand movements [30]. In a similar vein, Chae et al [31] proposed a VR based hand gesture identification system that used a density based CNN. The visualisation connected with trajectory reconstruction has been demonstrated in Amar et al [32], and lastly, nature grasping has been revealed in Tian et al [33].…”

Leukocytes identification using augmentation and transfer learning based convolution neural network

“…This study depends on the weights of the transfer method that learns from shared weights among the source and target of pre-trained models. Therefore, it can be used to deliver more improvements in classification accuracy as well as reduce the total training interval [30]- [32].…”