Interpretation of Swedish Sign Language Using Convolutional Neural Networks and Transfer Learning

Abstract

The automatic interpretation of sign languages is a challenging task, as it requires the usage of high-level vision and high-level motion processing systems for providing accurate image perception. In this paper, we use Convolutional Neural Networks (CNNs) and transfer learning to make computers able to interpret signs of the Swedish Sign Language (SSL) hand alphabet. Our model consists of the implementation of a pre-trained InceptionV3 network, and the usage of the mini-batch gradient descent optimization algorithm. We rely on transfer learning during the pre-training of the model and its data. The final accuracy of the model, based on 8 study subjects and 9400 images, is 85%. Our results indicate that the usage of CNNs is a promising approach to interpret sign languages, and transfer learning can be used to achieve high testing accuracy despite using a small training dataset. Furthermore, we describe the implementation details of our model to interpret signs as a user-friendly web application.

Interpretation of Swedish Sign Language using Convolutional Neural Networks and Transfer Learning

“…All code and data for this investigation is stored in three different GitHub repositories where each repository contains a README file with more information of its content. The code for the model, data generator, and front end for the application, can be found in one GitHub repository 12 . The code for the back end API for the application is stored separately and can be found another GitHub repository 13 .…”

“…This work is based on the Bachelor thesis of Peterson and Halvardsson [12], which was conducted in collaboration with Prevas AB, a Swedish technical IT consulting firm focusing on several areas of industry such as energy, defence, and life science. It has also been partially supported by the Wallenberg Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation.…”

Word separation in continuous sign language using isolated signs and post-processing

“…Here, we briefly review recent works in CSLR considering transfer learning. With the advent of deep learning in recent years, many recent approaches achieved state-of-the-art performance using the combination of different deep learning-based models, such as CNN and RNN [4,5,20,25,22,23,16,17,21,1,9,2,14,15,27,6,8,11,26,10,19,18]. More specifically, while many advancements have been obtained in ISLR and CSLR with the capabilities of deep learning-based models, some challenges in both tasks still need to be discussed.…”

On the Feasibility of Privacy-Secured Facial Authentication for low-power IoT Devices - Quantifying the Effects of Head Pose Variation on End-to-End Neural Face Recognition

“…To address these issues, the use of transfer learning for sign language recognition has been proposed [Halvardsson et al, 2021, Mocialov et al, 2020. Other works have proposed retraining the higher layers of common pre-trained convolutional neural networks (CNN) using synthetic imagery for an object recognition task [Rajpura et al, 2018, Carneiro et al, 2021.…”