Graph-Based Semisupervised Learning for Acoustic Modeling in Automatic Speech Recognition

Abstract

Graph-based semi-supervised learning (SSL) is a widely used semi-supervised learning method in which the labeled data and unlabeled data are jointly represented as a weighted graph, and the information is propagated from the labeled data to the unlabeled data. The key assumption that graph-based SSL makes is that data samples lie on a low dimensional manifold, where samples that are close to each other are expected to have the same class label. More importantly, by exploiting the relationship between training and test samples, graph-based SSL implicitly adapts to the test data.In this thesis, we address several key challenges in applying graph-based SSL to acoustic modeling. We first investigate and compare several state-of-the-art graph-based SSL algorithms on a benchmark dataset. In addition, we propose novel graph construction methods that allow graph-based SSL to handle variable-length input features. We next investigate the efficacy of graph-based SSL in context of a fully-fledged DNN-based ASR system. We compare two different integration frameworks for graph-based learning. First, we propose a lattice-based late integration framework that combines graph-based SSL with the DNN-based acoustic modeling and evaluate the framework on continuous word recognition tasks. Second, we propose an early integration framework using neural graph embeddings and compare two different neural graph embedding features that capture the information of the manifold at different levels. The embedding features are used as input to a DNN system and are shown to outperform the conventional acoustic feature inputs on several medium-to-large vocabulary conversational speech recognition tasks.

Dynamic Emotion Modeling With Learnable Graphs and Graph Inception Network

“…GCNs have been used to address skeleton-based action recognition recorded using motion capture [13]. The application of graph networks has also started emerging in automatic speech recognition [18].…”

Automatic Speech Recognition: Systematic Literature Review

“…In [20], the authors compared several graph-based algorithms and proposed the prior-regularized measure propagation (pMP) algorithm. They evaluated two different frameworks for integrating graph-based learning into state-of-the-art DDN-based speech recognition systems.…”

“…In [20], the authors investigated graph-based semisupervised learning (SSL) in DNN-based acoustic models for speech recognition. They compared several graph-based learning (GBL) algorithms and proposed the pMP algorithm.…”

Modeling Latent Topics and Temporal Distance for Story Segmentation of Broadcast News

“…We have seen the use of graph Laplacians [35] in various tasks in machine learning, including clustering [36], [37], manifold learning [38] and semi-supervised learning [39], [40]. Graph Laplacians are widely used to preserve the local geometric structure of data in an optimization task.…”