Discriminative segmental cascades for feature-rich phone recognition

Abstract

Discriminative segmental models, such as segmental conditional random fields (SCRFs) and segmental structured support vector machines (SSVMs), have had success in speech recognition via both lattice rescoring and first-pass decoding. However, such models suffer from slow decoding, hampering the use of computationally expensive features, such as segment neural networks or other high-order features. A typical solution is to use approximate decoding, either by beam pruning in a single pass or by beam pruning to generate a lattice followed by a second pass. In this work, we study discriminative segmental models trained with a hinge loss (i.e., segmental structured SVMs). We show that beam search is not suitable for learning rescoring models in this approach, though it gives good approximate decoding performance when the model is already well-trained. Instead, we consider an approach inspired by structured prediction cascades, which use max-marginal pruning to generate lattices. We obtain a highaccuracy phonetic recognition system with several expensive feature types: a segment neural network, a second-order language model, and second-order phone boundary features.Index Terms-segmental conditional random field, structured prediction cascades, phone recognition, segment neural network, beam search

End-to-End Neural Segmental Models for Speech Recognition

“…The multi-stage training results are shown in Table I. The results are consistent with those reported in [17]. For the (PYRAMID OR NOT) ARE TRAINED WITH THE FRAME-WISE CROSS ENTROPY.…”

“…The first type of weight function, termed the frame classifier (FC) weight, is similar to weight functions used in a variety of prior work [15], [17], [16]. A frame classifier takes in the LSTM output h 1 , .…”

“…Tang et al first compared different losses and training strategies for segmental models in a rescoring framework [27] and then in first-pass segmental models [26]. [42] 16.5 SUMMIT (1998) [43], [13] 24.4 segmental CRF (SCRF) (2012) [14] 33.1 SCRF + shallow NN (2012) [15] 26.5 SCRF + DNN (2015) [41] 19.1 deep segmental NN (2013) [16] 21.9 segmental cascades (2015) [17] 19.9 segmental RNN (SRNN) (2016) [18] 18.9 17.3 end-to-end + two-stage training (2016) [26] 19.7 SRNN + multitask (2017) [30] 18.7 17.5 SRNN + multitask (2017) (this work) 18.5 They also introduced segment-level classifiers and segmental cascades for incorporating them (and other expensive features) into segmental weight functions [17]. Lu et al [18] introduced an LSTM-based weight function for every segment, and were also the first to use pyramid LSTMs to speed up inference for segmental models.…”

“…Segmental models that operate on the full search space-first-pass segmental models-have not been explored until recently [14]. Since then, there has been a variety of work exploring better segment representations for first-pass segmental models, especially ones that depend on neural networks [15], [16], [17], [18].…”

“…In this work, we will show results comparing segmental models trained in multiple stages with intermediate supervision to ones trained end to end. In this context, we explore two weight functions, one based on frame classifiers [15], [17], [16], and one based on segmental recurrent neural networks [22], [18]. We will also compare end-to-end frame-based and segmental models in terms of their search spaces and loss functions.…”