Deep Learning Sign Language Recognition System Based on Wi-Fi CSI

Abstract

Many sensing gesture recognition systems based on Wi-Fi signals are introduced because of the commercial off-the-shelf Wi-Fi devices without any need for additional equipment. In this paper, a deep learning-based sign language recognition system is proposed. Wi-Fi CSI amplitude and phase information is used as input to the proposed model. The proposed model uses three types of deep learning: CNN, LSTM, and ABLSTM with a complete study of the impact of optimizers, the use of amplitude and phase of CSI, and preprocessing phase. Accuracy, F-score, Precision, and recall are used as performance metrics to evaluate the proposed model. The proposed model achieves 99.855%, 99.674%, 99.734%, and 93.84% average recognition accuracy for the lab, home, lab + home, and 5 different users in a lab environment, respectively. Experimental results show that the proposed model can effectively detect sign gestures in complex environments compared with some deep learning recognition models.

A Multitask Sign Language Recognition System Using Commodity Wi-Fi

“…SignFi [3] achieved the average recognition accuracy of 98.01%, 98.91%, 94.81%, and 86.66% in the lab276, home276, lab + home276, and lab150, respectively. In 2020, reference [31] compared the three types of deep learning: long shortterm memory (LSTM), CNN, and attentive bi-directional LSTM (ABLSTM). Te experimental results showed that the CNN model had the best recognition performance on the SignFi dataset.…”

“…We obtained the raw CSI data from the SignFi dataset in Figure 2. Te magnitude and phase of each raw CSI sample can be extracted, normalized, and transformed into a CSI image of size 200 × 60 × 30 as described in [3,31]. Unlike the abovementioned literature, we do not denoise and unwrap the amplitude and phase information.…”

“…Te other methods are all CNN methods. Te input data modality of the Wi-SignFi model is the same as that of CNN [31] and SignFi [3], both are concatenations of amplitude and phase information. Te input data of dual-output twostream with ResNet50 [4] contains not only amplitude and phase information, but also diference information including the gesture motion.…”

“…Te input data for the abovementioned CNN models are all preprocessed, except for the Wi-SignFi model. Te input data resolution of CNN [31] and SignFi [3] are both 200 × 60 × 3, while the dual-output two-stream with ResNet50 [4] is 224 × 224 × 3. CNN [31] achieved the best recognition results in Lab276 and Lab + Home276, 99.855% and 99.73%, respectively.…”

“…Te input data resolution of CNN [31] and SignFi [3] are both 200 × 60 × 3, while the dual-output two-stream with ResNet50 [4] is 224 × 224 × 3. CNN [31] achieved the best recognition results in Lab276 and Lab + Home276, 99.855% and 99.73%, respectively. However, Wi-SignFi outperforms other methods in recognition accuracy on the Home276 and Lab150 datasets, 99.75% and 96.41%, respectively.…”