Speech control of robotic hand augmented with 3D animation using neural network

Ismail, Rifky; Ariyanto, Mochammad; Caesarendra, Wahyu; Haryanto, Ismoyo; Dewoto, Hadianto K.; Paryanto,

doi:10.1109/iecbes.2016.7843469

Cited by 5 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For fast transmission data between SRF and host computer, the selected baud rate is 115.200 bit/s. Based on the previous research, the virtual hand or 3D animation of robotic hand can run simultaneously with robotic hand [6,7]. The 3D CAD model of SRF as shown in figure 1 (a) is exported from SolidWorks software to SimMechanics First Generation.…”

Section: Srf With 3d Animationmentioning

confidence: 99%

Development of 6 DOF Supernumerary Robotic Fingers Integrated with 3D Animation

Ariyanto

Setiawan

Ismail

et al. 2018

MATEC Web of Conferences

Self Cite

View full text Add to dashboard Cite

The paper presents the development of wearable robot called supernumerary robotic fingers (SRF) that can be integrated with 3D Animation. The robot comprises of two parts, robotic finger (RF) and robotic thumb (RT). RF is located next to little finger, and RT is located next to the thumb. The robot is driven by modified glove sensor. 3D CAD model is exported into SimMechanics for 3D Animation purpose under MATLAB/Simulink environment. Forward kinematics of the robot is studied using SimMechanics and the trajectory of each robotic fingertip will be calculated. Based on the experimental result, the robot and its 3D animation can be operated by a healthy person in real time. The robot can assist in object manipulation with ten-inch tablet using single human hand.

show abstract

Section: Srf With 3d Animationmentioning

confidence: 99%

Development of 6 DOF Supernumerary Robotic Fingers Integrated with 3D Animation

Ariyanto

Setiawan

Ismail

et al. 2018

MATEC Web of Conferences

Self Cite

View full text Add to dashboard Cite

show abstract

“…Electric powered prosthetic hands can receive commands from users by one or few ways such as push-button, joystick, keyboard, text, electroencephalography (EEG) [9], Electroneurography (ENG) [10][11][12], electromyography (EMG) [13][14][15][16][17][18][19][20], vision [21][22][23], and speech [24][25][26][27][28]. Among these ways, electromyography is the most convenient for amputees.…”

Section: Introductionmentioning

confidence: 99%

“…Ismail et al designed a speech-based controller for a robotic hand using 13 selected features that comprise eight features in the frequency domain and five features in the time domain along with a multilayer perceptron (MLP). They performed training and testing in an identical room to reduce the influence of external noise [24]. Vijayaragavan et al used Google speech-to-text to control prosthetic hand by android devices [26].…”

Section: Introductionmentioning

confidence: 99%

End-to-End Learning of Speech 2D Feature-Trajectory for Prosthetic Hands

Jafarzadeh

Tadesse

2020

2020 Second International Conference on Transdisciplinary AI (TransAI)

View full text Add to dashboard Cite

Speech is one of the most common forms of communication in humans. Speech commands are essential parts of multimodal controlling of prosthetic hands. In the past decades, researchers used automatic speech recognition systems for controlling prosthetic hands by using speech commands. Automatic speech recognition systems learn how to map human speech to text. Then, they used natural language processing or a look-up table to map the estimated text to a trajectory. However, the performance of conventional speech-controlled prosthetic hands is still unsatisfactory. Recent advancements in generalpurpose graphics processing units (GPGPUs) enable intelligent devices to run deep neural networks in real-time. Thus, architectures of intelligent systems have rapidly transformed from the paradigm of composite subsystems optimization to the paradigm of end-to-end optimization. In this paper, we propose an end-to-end convolutional neural network (CNN) that maps speech 2D features directly to trajectories for prosthetic hands. The proposed convolutional neural network is lightweight, and thus it runs in real-time in an embedded GPGPU. The proposed method can use any type of speech 2D feature that has local correlations in each dimension such as spectrogram, MFCC, or PNCC. We omit the speech to text step in controlling the prosthetic hand in this paper. The network is written in Python with Keras library that has a TensorFlow backend. We optimized the CNN for NVIDIA Jetson TX2 developer kit. Our experiment on this CNN demonstrates a root-mean-square error of 0.119 and 20ms running time to produce trajectory outputs corresponding to the voice input data. To achieve a lower error in real-time, we can optimize a similar CNN for a more powerful embedded GPGPU such as NVIDIA AGX Xavier.

show abstract

“…To control a hand exoskeleton, developers used a combination of discrete wavelet transforms and hidden Markov models [28]. Developers used a multi-layer perceptron to control a robotic hand with 13 speech features that compromise five features in the time domain and eight features in the frequency domain [29]. To reduce the influence of external noise, they performed training and testing in an identical room.…”

Section: Introductionmentioning

confidence: 99%

Convolutional Neural Networks for Speech Controlled Prosthetic Hands

Jafarzadeh

Tadesse

2019

2019 First International Conference on Transdisciplinary AI (TransAI)

View full text Add to dashboard Cite

Speech recognition is one of the key topics in artificial intelligence, as it is one of the most common forms of communication in humans. Researchers have developed many speech-controlled prosthetic hands in the past decades, utilizing conventional speech recognition systems that use a combination of neural network and hidden Markov model. Recent advancements in general-purpose graphics processing units (GPGPUs) enable intelligent devices to run deep neural networks in real-time. Thus, state-of-the-art speech recognition systems have rapidly shifted from the paradigm of composite subsystems optimization to the paradigm of end-to-end optimization. However, a low-power embedded GPGPU cannot run these speech recognition systems in real-time. In this paper, we show the development of deep convolutional neural networks (CNN) for speech control of prosthetic hands that run in real-time on a NVIDIA Jetson TX2 developer kit. First, the device captures and converts speech into 2D features (like spectrogram). The CNN receives the 2D features and classifies the hand gestures. Finally, the hand gesture classes are sent to the prosthetic hand motion control system. The whole system is written in Python with Keras, a deep learning library that has a TensorFlow backend. Our experiments on the CNN demonstrate the 91% accuracy and 2ms running time of hand gestures (text output) from speech commands, which can be used to control the prosthetic hands in real-time.

show abstract

Speech control of robotic hand augmented with 3D animation using neural network

Cited by 5 publications

References 14 publications

Development of 6 DOF Supernumerary Robotic Fingers Integrated with 3D Animation

Development of 6 DOF Supernumerary Robotic Fingers Integrated with 3D Animation

End-to-End Learning of Speech 2D Feature-Trajectory for Prosthetic Hands

Convolutional Neural Networks for Speech Controlled Prosthetic Hands

Contact Info

Product

Resources

About