Virtual guitar: using real-time finger tracking for musical instruments

Hakim, Noorkholis Luthfil; Sun, Shih Wei; Hsu, Mu Hsen; Shih, Timothy K.; Wu, Shih Jung

doi:10.1504/ijcse.2019.099081

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors argue that the use of music games can expand the conceptualization of what is "playful" and "serious" as opposed to traditional teaching. Experiments have also been run using a virtual musical instrument system (using Kinect) as an initiation method for someone who does not own the real instrument [24].Furthermore, augmented reality (AR) has been used to mitigate the limitations of the different didactic materials used to learn guitar [18,25]. This technology has been used in education for several reasons, including reduced costs and the improvement of hardware/software, the need for…”

mentioning

confidence: 99%

See 1 more Smart Citation

AR Graphic Representation of Musical Notes for Self-Learning on Guitar

et al. 2019

View full text Add to dashboard Cite

Despite being one of the most commonly self-taught instruments, and despite the ready availability of significant amounts of didactic material, the guitar is a challenging instrument to learn. This paper proposes an application based on augmented reality (AR) that is designed to teach beginner students basic musical chords on the guitar, and provides details of the experimental study performed to determine whether the AR methodology produced faster results than traditional one-on-one training with a music teacher. Participants were divided into two groups of the same size. Group 1 consisted of 32 participants who used the AR app to teach themselves guitar, while Group 2, with a further 32 participants, received formal instruction from a music teacher. Results found no differences in learning times between the two groups based on the variables of method and gender. However, participant feedback suggested that there are advantages to the self-taught approach using AR that are worth considering. A system usability scale (SUS) questionnaire was used to measure the usability of the application, obtaining a score of 82.5, which was higher than the average of 68 that indicates an application to be good from a user experience point of view, and satisfied the purpose for which the application was created.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

AR Graphic Representation of Musical Notes for Self-Learning on Guitar

et al. 2019

View full text Add to dashboard Cite

show abstract

“…It has inspired new technologies such as gesture-based signaling systems [5,6], virtual reality [7,8], and smart television [9,10] in the computer-vision and design recognition industries. Numerous applications have made significant progress, including the recognition of sign language [11,12], the control of robots [13,14], and the playing of virtual musical instruments [15,16]. The progress in recognizing gestures by hands has also attracted much attention from the industrial sector in manufacturing interactions between human robots [17,18] and self-driving cars [19].…”

Section: Introductionmentioning

confidence: 99%

Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach

et al. 2022

View full text Add to dashboard Cite

Hearing-impaired people use sign language to express their thoughts and emotions and reinforce information delivered in daily conversations. Though they make a significant percentage of any population, the majority of people can't interact with them due to limited or no knowledge of sign languages. Sign language recognition aims to detect the significant motions of the human body, especially hands, analyze them and understand them. Such systems may become life-saving when hearing-challenged people are in desperate situations like heart attacks, accidents, etc. In the present study, deep learning-based hand gesture recognition models are developed to accurately predict the emergency signs of Indian Sign Language (ISL). The dataset used contains the videos for eight different emergency situations. Several frames were extracted from the videos and are fed to three different models. Two models are designed for classification, while one is an object detection model, applied after annotating the frames. The first model consists of a three-dimensional convolutional neural network (3D CNN), while the second comprises of a pre-trained VGG-16 and a recurrent neural network with a long short-term memory (RNN-LSTM) scheme. The last model is based on YOLO (You Only Look Once) v5, an advanced object detection algorithm. The prediction accuracies of the classification models were 82% and 98%, respectively. YOLO based model outperformed the rest and achieved an impressive mean average precision of 99.6%.

show abstract

“…In recent years, hand gesture recognition has inspired new technologies in the computer vision and pattern recognition fields, such as Virtual reality [1,2] and Smart TV or interactive system [3,4]. Significant progress of this field has been accomplished in many applications, i.e., sign language recognition [5,6], robot control [7,8], virtual musical instrument performance [9,10].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model

Hakim

Shih

Arachchi

et al. 2019

Sensors

Self Cite

View full text Add to dashboard Cite

With the recent growth of Smart TV technology, the demand for unique and beneficial applications motivates the study of a unique gesture-based system for a smart TV-like environment. Combining movie recommendation, social media platform, call a friend application, weather updates, chatting app, and tourism platform into a single system regulated by natural-like gesture controller is proposed to allow the ease of use and natural interaction. Gesture recognition problem solving was designed through 24 gestures of 13 static and 11 dynamic gestures that suit to the environment. Dataset of a sequence of RGB and depth images were collected, preprocessed, and trained in the proposed deep learning architecture. Combination of three-dimensional Convolutional Neural Network (3DCNN) followed by Long Short-Term Memory (LSTM) model was used to extract the spatio-temporal features. At the end of the classification, Finite State Machine (FSM) communicates the model to control the class decision results based on application context. The result suggested the combination data of depth and RGB to hold 97.8% of accuracy rate on eight selected gestures, while the FSM has improved the recognition rate from 89% to 91% in a real-time performance.

show abstract

Virtual guitar: using real-time finger tracking for musical instruments

Cited by 4 publications

References 0 publications

AR Graphic Representation of Musical Notes for Self-Learning on Guitar

AR Graphic Representation of Musical Notes for Self-Learning on Guitar

Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach

Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model

Contact Info

Product

Resources

About