Malaysian Sign Language Recognition Using 3D Hand Pose Estimation

Abstract

Sign languages are one of those mediums for hearing-impaired people. These languages transmit meaning by visual-manual treatment, or more simply, hand movement. Currently, there are only 95 sign language interpreters registered with the Malaysian Federation of the Deaf as of 2020, compared to 40,389 hearing-impaired individuals with disabilities registered with the welfare department which is a problem. Therefore, with the use of deep-learning technology, this paper proposes to alleviate the scarcity of Malaysian Sign Language interpreters for the benefit of hearing-impaired persons. The paper aims to test and report a sequenced 3D keypoint hand pose estimation model for Malaysian Sign Language Recognition and evaluate the implementation of action model in decoding basic poses of Malaysian Sign Language. According to the findings, the detecting of 3D keypoints and incorporating into LSTM models using deep learning machine learning platform and framework like TensorFlow and MediaPipe enables the detection of Malaysian sign language 3D hand posture estimation. The results demonstrated that 3D hand posture estimation may be utilised to estimate sign language in real time, providing for a better interpretation approach for the deaf community.