AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

Abstract

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains color image (RGB), depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used Convolutional Neural Networks (CNNs) to extract features, unidirectional and bidirectional Long Short-Term Memory (LSTM) models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

Using Motion History Images With 3D Convolutional Networks in Isolated Sign Language Recognition

“…Then, they built a temporal attention-based model for classification. In our previous work [5], we proposed a baseline model for a new large-scale isolated Turkish Sign Language (AUTSL) dataset. We integrated a Feature Pooling Module and a temporal attention model to focus on more relevant spatio-temporal parts of the videos.…”

“…We evaluate our proposed models on two very recently shared large-scale isolated sign language datasets: AUTSL [5] and BosphorusSign22k [6].…”

“…AUTSL [5] is a large-scale, signer independent, isolated TSL dataset that contains 226 signs, and 36,302 video samples. The dataset contains 43 signers; 31 of them are included in the training set, 6 are in the validation set, and the remaining 6 are in the test set.…”

Artificial Intelligence Technologies for Sign Language

“…Contrary to the previous methods that use a single Kinect sensor, this work additionally employs a machine vision camera, along with a television screen, for sign demonstration. Sincan et al in [ 16 ], captured isolated Turkish sign language glosses using Kinect sensors with a large variety of indoor and outdoor backgrounds, revealing the importance of capturing videos with various backgrounds. Adaloglou et al in [ 17 ], created a large sign language dataset with RealSense D435 sensor that records both RGB and depth information.…”

Analytical review of models and methods for automatic recognition of gestures and sign languages

“…Примечание: ЗР -задача распознавания; ПО -количество предметных областей; Р -разметка; УЗ -устройство захвата (камера, сенсор); ЦК -цветная камера; н/д -нет данных.ИНФОРМАЦИОННОУПРАВЛЯЮЩИЕ СИСТЕМЫ № 6, 202112 …”