Dynamic GAN for High-Quality Sign Language Video Generation from Skeletal poses using Generative Adversarial Networks

Abstract

The emergence of unsupervised generative models has resulted in greater performance in image and video generation tasks. However, existing generative models pose huge challenges in high-quality video generation process due to blurry and inconsistent results. In this paper, we introduce a novel generative framework named Dynamic Generative Adversarial Networks (Dynamic GAN) model for regulating the adversarial training and generating photorealistic high-quality sign language videos from skeletal poses. The proposed model comprises three stages of development such as generator network, classification and image quality enhancement and discriminator network. In the generator fold, the model generates samples similar to real images using random noise vectors, the classification of generated samples are carried out using the VGG-19 model and novel techniques are employed for improving the quality of generated samples in the second fold of the model and finally the discriminator networks fold identifies the real or fake samples. Unlike, existing approaches the proposed novel framework produces photo-realistic video quality results without using any animation or avatar approaches. To evaluate the model performance qualitatively and quantitatively, the proposed model has been evaluated using three benchmark datasets that yield plausible results. The datasets are RWTH-PHOENIX-Weather 2014T dataset, and our self-created dataset for Indian Sign Language (ISL-CSLTR), and the UCF-101 Action Recognition dataset. The output samples and performance metrics show the outstanding performance of our model.

Development of an End-to-End Deep Learning Framework for Sign Language Recognition, Translation, and Video Generation

“…To vectorize the output as a single array, the fully connected layer is used. The incorporation of dynamic GAN [86] provides high quality video generation results by encompassing the various approaches such as frame generation and video completion techniques. The LSTM network is used for predicting the text equivalents of the sign gestures and further helps to produce the language sentences.…”