End-of-Utterance Prediction by Prosodic Features and Phrase-Dependency Structure in Spontaneous Japanese Speech

Abstract

This study is aimed at uncovering a way that participants in conversation predict end-of-utterance for spontaneous Japanese speech. In spontaneous everyday conversation, the participants must predict the ends of utterances of a speaker to perform smooth turn-taking without too much gap. We consider that they utilize not only syntactic factors but also prosodic factors for the end-of-utterance prediction because of the difficulty of prediction of a syntactic completion point in spontaneous Japanese. In previous studies, we found that prosodic features changed significantly in the final accentual phrase. However, it is not clear what prosodic features support the prediction. In this paper, we focused on dependency structure among bunsetsuphrases as the syntactic factor, and investigated the relation between the phrase-dependency and prosodic features. The results showed that the average fundamental frequency and the average intensity for accentual phrases did not decline until the modified phrase appeared. Next, to predict the end of utterance from the syntactic and prosodic features, we constructed a generalized linear mixed model. The model provided higher accuracy than using the prosodic features only. These suggest the possibility that prosodic changes and phrase-dependency relations inform the hearer that the utterance is approaching its end.

Prediction of Turn-taking Using Multitask Learning with Prediction of Backchannels and Fillers

“…There are some works that investigated other features in speech such as N-gram model [13], dependency structures [14], and the previous turn-taking behaviors [15]. There are also other works that investigated non-verbal features such as respiratory features [16], head pose features [17], and eye-gaze features [18].…”

Improving End-of-Turn Detection in Spoken Dialogues by Detecting Speaker Intentions as a Secondary Task

“…One line of work looked at the use of acoustic and lexical features for modeling turn-taking behavior [4,5,9,10]. Liu et al [5], Masumura et al [9], and Ishimoto et al [10] looked at the problem in Japanese conversations while Maier et al [4] looked at the problem in German conversations. Masumura et al proposed using stacked time-asynchronous sequential networks for detecting end-of-turns given sequences of asynchronous features (e.g., MFCCs and words) [9].…”

“…One line of work looked at the use of acoustic and lexical features for modeling turn-taking behavior [4,5,9,10]. Liu et al [5], Masumura et al [9], and Ishimoto et al [10] looked at the problem in Japanese conversations while Maier et al [4] looked at the problem in German conversations.…”

“…Masumura et al proposed using stacked time-asynchronous sequential networks for detecting end-of-turns given sequences of asynchronous features (e.g., MFCCs and words) [9]. Ishimoto et al investigated the dependency between syntactic and prosodic features and showed that combining the two features is useful for predicting end-of-turns [10]. Liu et al built a Recurrent Neural Network (RNN) to classify a given utterance into four classes that relate to turn-taking behavior using joint acoustic and lexical embeddings [5].…”

Turn-Taking Prediction Based on Detection of Transition Relevance Place

“…The majority of investigated feature sets are based on prosodic features such as fundamental frequency (F0) and power [9,10,11,12,13]. Linguistic features were also investigated such as syntactic structure, turn-ending markers, and language model [14,15]. Moreover, multi-modal features were also considered such as eye-gaze [16,17,18,19], respiration [20,21,22], and head-direction [16,23].…”