Spoken Language Understanding of Human-Machine Conversations for Language Learning Applications

Ye, Qian; Ubale, Rutuja; Lange, Patrick; Evanini, Keelan; Ramanarayanan, Vikram; Soong, Frank K.

doi:10.1007/s11265-019-01484-3

Cited by 8 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the provisions of information theory, we argue the solution rule (2) in the context of the relative entropy functional [ 35 , 36 , 37 ] (3):

where

is the selective probability distribution of the studied (empirical) speech signal

relative to the etalon probability distribution

. Assume that the distribution law

is normal:

, where

is a sample matrix of autocorrelation of the speech signal

of dimension

.…”

Section: Methodsmentioning

confidence: 99%

“…Let

, i.e., two competing hypotheses,

and

, are tested for a priori unknown autocorrelation matrices

and

. The verification will be performed using the asymptotic minimax criterion of the likelihood ratio [ 35 , 36 , 37 ] based on data from a sample

. Under such conditions, the hypothesis

will be considered true if the condition

is satisfied, where

is the plausibility function of the signal

provided that hypothesis

is confirmed, and

is the plausibility function of the signal

.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Entropy-Argumentative Concept of Computational Phonetic Analysis of Speech Taking into Account Dialect and Individuality of Phonation

Kovtun

Семенов

2022

Entropy

View full text Add to dashboard Cite

In this article, the concept (i.e., the mathematical model and methods) of computational phonetic analysis of speech with an analytical description of the phenomenon of phonetic fusion is proposed. In this concept, in contrast to the existing methods, the problem of multicriteria of the process of cognitive perception of speech by a person is strictly formally presented using the theoretical and analytical apparatus of information (entropy) theory, pattern recognition theory and acoustic theory of speech formation. The obtained concept allows for determining reliably the individual phonetic alphabet inherent in a person, taking into account their inherent dialect of speech and individual features of phonation, as well as detecting and correcting errors in the recognition of language units. The experiments prove the superiority of the proposed scientific result over such common Bayesian concepts of decision making using the Euclidean-type mismatch metric as a method of maximum likelihood and a method of an ideal observer. The analysis of the speech signal carried out in the metric based on the proposed concept allows, in particular, for establishing reliably the phonetic saturation of speech, which objectively characterizes the environment of speech signal propagation and its source.

show abstract

“…Based on the provisions of information theory, we argue the solution rule (2) in the context of the relative entropy functional [ 35 , 36 , 37 ] (3):

where

is the selective probability distribution of the studied (empirical) speech signal

relative to the etalon probability distribution

. Assume that the distribution law

is normal:

, where

is a sample matrix of autocorrelation of the speech signal

of dimension

.…”

Section: Methodsmentioning

confidence: 99%

“…Let

, i.e., two competing hypotheses,

and

, are tested for a priori unknown autocorrelation matrices

and

. The verification will be performed using the asymptotic minimax criterion of the likelihood ratio [ 35 , 36 , 37 ] based on data from a sample

. Under such conditions, the hypothesis

will be considered true if the condition

is satisfied, where

is the plausibility function of the signal

provided that hypothesis

is confirmed, and

is the plausibility function of the signal

.…”

Section: Methodsmentioning

confidence: 99%

Entropy-Argumentative Concept of Computational Phonetic Analysis of Speech Taking into Account Dialect and Individuality of Phonation

Kovtun

Семенов

2022

Entropy

View full text Add to dashboard Cite

show abstract

“…SLU task aims to extract structured semantic representations from speech signals [3,4]. Conventional cascaded SLU systems consist of an automatic speech recognition (ASR) module as well as a downstream natural language understanding (NLU) module [5]. On the other hand, end-to-end (E2E) SLU systems directly extract users' intentions from input speech to avoid error propagation seen in the above cascaded method [1,6].…”

Section: Introductionmentioning

confidence: 99%

Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation

Deng¹,

Watanabe²,

Shi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Although Transformers have gained success in several speech processing tasks like spoken language understanding (SLU) and speech translation (ST), achieving online processing while keeping competitive performance is still essential for real-world interaction. In this paper, we take the first step on streaming SLU and simultaneous ST using a blockwise streaming Transformer, which is based on contextual block processing and blockwise synchronous beam search. Furthermore, we design an automatic speech recognition (ASR)-based intermediate loss regularization for the streaming SLU task to improve the classification performance further. As for the simultaneous ST task, we propose a cross-lingual encoding method, which employs a CTC branch optimized with target language translations. In addition, the CTC translation output is also used to refine the search space with CTC prefix score, achieving joint CTC/attention simultaneous translation for the first time. Experiments for SLU are conducted on FSC and SLURP corpora, while the ST task is evaluated on Fisher-CallHome Spanish and MuST-C En-De corpora. Experimental results show that the blockwise streaming Transformer achieves competitive results compared to offline models, especially with our proposed methods that further yield a 2.4% accuracy gain on the SLU task and a 4.3 BLEU gain on the ST task over streaming baselines.

show abstract

“…These representations are crucial to enable speech as the primary mode of human-computer interaction (HCI) [2]. Traditional SLU solutions rely on the text transcription generated by an automatic speech recognition (ASR) module, followed by a natural language understanding (NLU) system, responsible for extracting semantics from the ASR output [3]. As described in [4], in such scenarios the ASR typically operates on chunks of the incoming speech signal and outputs the transcript for each segment.…”

Section: Introductionmentioning

confidence: 99%

Sequential End-to-End Intent and Slot Label Classification and Localization

Cao

Potdar

Avila

2021

Preprint

View full text Add to dashboard Cite

Human-computer interaction (HCI) is significantly impacted by delayed responses from a spoken dialogue system. Hence, endto-end (e2e) spoken language understanding (SLU) solutions have recently been proposed to decrease latency. Such approaches allow for the extraction of semantic information directly from the speech signal, thus bypassing the need for a transcript from an automatic speech recognition (ASR) system. In this paper, we propose a compact e2e SLU architecture for streaming scenarios, where chunks of the speech signal are processed continuously to predict intent and slot values. Our model is based on a 3D convolutional neural network (3D-CNN) and a unidirectional long short-term memory (LSTM). We compare the performance of two alignment-free losses: the connectionist temporal classification (CTC) method and its adapted version, namely connectionist temporal localization (CTL). The latter performs not only the classification but also localization of sequential audio events. The proposed solution is evaluated on the Fluent Speech Command dataset and results show our model ability to process incoming speech signal, reaching accuracy as high as 98.97 % for CTC and 98.78 % for CTL on single-label classification, and as high as 95.69 % for CTC and 95.28 % for CTL on two-label prediction.

show abstract

Spoken Language Understanding of Human-Machine Conversations for Language Learning Applications

Cited by 8 publications

References 29 publications

Entropy-Argumentative Concept of Computational Phonetic Analysis of Speech Taking into Account Dialect and Individuality of Phonation

Entropy-Argumentative Concept of Computational Phonetic Analysis of Speech Taking into Account Dialect and Individuality of Phonation

Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation

Sequential End-to-End Intent and Slot Label Classification and Localization

Contact Info

Product

Resources

About