Harnessing GANs for Zero-Shot Learning of New Classes in Visual Speech Recognition

Abstract

Visual Speech Recognition (VSR) is the process of recognizing or interpreting speech by watching the lip movements of the speaker. Recent machine learning based approaches model VSR as a classification problem; however, the scarcity of training data leads to error-prone systems with very low accuracies in predicting unseen classes. To solve this problem, we present a novel approach to zero-shot learning by generating new classes using Generative Adversarial Networks (GANs), and show how the addition of unseen class samples increases the accuracy of a VSR system by a significant margin of 27% and allows it to handle speaker-independent out-of-vocabulary phrases. We also show that our models are language agnostic and therefore capable of seamlessly generating, using English training data, videos for a new language (Hindi). To the best of our knowledge, this is the first work to show empirical evidence of the use of GANs for generating training samples of unseen classes in the domain of VSR, hence facilitating zero-shot learning. We make the added videos for new classes publicly available along with our code1.

A Survey of Long‐Tail Item Recommendation Methods

“…In 2020, the long-tail item recommendation method began to the deep learning method. For example, Bai et al [36] use stacked denoising autoencoders (SDAE) to realize online long-tail item recom-mendation, and Kumar et al [12] proposed to realize longtail item recommendation using few shot learning. Bai et al [36] proposed a deep learning framework for long-tail item recommendation (DLTSR).…”

“…The content-based recommendation method and collaborative filtering recommendation method [2,3] are classic methods in the recommender system. Machine learning and deep learning have great advantages in learning the inherent laws and representation levels of sample data and have made many research achievements in image classification [4][5][6][7], object detection [8][9][10][11], speech recognition [12,13], and emotion recognition [14]. Therefore, researchers combine machine learning, deep learning, knowledge graph, and other technologies in these basic methods, allowing recommender systems to be widely used in news, tourism, e-commerce, and other fields.…”

LIFI: Towards Linguistically Informed Frame Interpolation

“…A related task to video frame interpolation is talking face generation. Here, given an audio waveform, the task is to synthesize a talking face [4,5,6]. In recent times, these approaches have become popular for both academic and non-academic purposes [7].…”

“…In recent times, these approaches have become popular for both academic and non-academic purposes [7]. While, on the one hand, they are being used to extend speechreading models to low resource languages [6], on the other, many of them are also used to generate fake news and paid content as well.…”

“…Speech as a natural signal is composed of three parts [10]: visual modality, audio modality, and the context in which it was spoken (crudely, the role played by language). Correspondingly, there are three tasks for modeling speech: speech-reading (or popularly known as lipreading) [6,11,12], speech recognition (or ASR) [13] and language modeling [14]. The part of speech which is closest to the speech video generation task is the visual modality of speech; and visemes are the fundamental units of this part of speech.…”