Expansion of WFST-Based Dialog Management for Handling Multiple ASR Hypotheses

Abstract

We proposed a weighted finite-state transducer-based dialog manager (WFSTDM) which was a platform for expandable and adaptable dialog systems. In this platform, all rules and/or models for dialog management (DM) are expressed in WFST form, and the WFSTs are used to accomplish various tasks via multiple modalities. With this framework, we constructed a statistical dialog system using the user concept and system action tags which were acquired from an annotated corpus of human-to-human spoken dialogs as input and output labels of the WFST. We introduced a spoken language understanding (SLU) WFST for converting user utterances to user concept tags, a dialog scenario WFST for converting user concept tags to system action tags and a sentence generation (SG) WFST for converging system action tags to system utterances. The tag sequence probabilities of the dialog scenario WFST were estimated by using a spoken dialog corpus for hotel reservation. The SLU, scenario and SG WFSTs were then composed to be a dialog management WFST which determines the next action of the system responding to the user input. In our previous research, we evaluated the dialog strategy by referring to the manual transcription. Then in this paper, we present the performance of WFSTDM when speech recognition hypotheses are input. To alleviate degradation of the DM performance caused by speech recognition errors, we expand the WFSTDM for handling multiple hypotheses of speech recognition and confidence score which indicate acoustic and linguistic reliability of speech recognition. We also evaluated the accuracy of SLU results and the correctness of system actions selected by the dialog management WFST. We confirmed that the performance of dialog management was enhanced by choosing the optimal action among all the WFST paths for multiple hypotheses (N-best) of speech recognition in consideration of confidence score.

Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models

“…In [12] a dialog system is described which transforms textual user utterances into response sentences using weighted FSTs, with the goal to be able to run a full back-and-forth dialog with the users. It was extended by [13] to accept n-best hypotheses from a triphone model acoustic model, which were combined with an additional 3-gram language model, as input. Eesen [14] introduced FST-decoding to models outputting character-based Connectionist Temporal Classification (CTC) [15] labels, similar to the Quartznet model of Scribosermo.…”

“…Alexa also uses FSTs for its skill kit, but keeps separate models for STT and NLU [16]. This work follows a very similar decoding approach as Eesen, which allows using recent CTC-based STT models (in difference to [10,11,13]), but alters the Grammar-FST (explained in the next chapters) to embed NLU information into it, similar to the semantic tagging of [9,10,11], which allows combining the two distinct STT+NLU models into a single SLU decoder.…”