A Discriminative Acoustic-Prosodic Approach for Measuring Local Entrainment

Abstract

Acoustic-prosodic entrainment describes the tendency of humans to align or adapt their speech acoustics to each other in conversation. This alignment of spoken behavior has important implications for conversational success. However, modeling the subtle nature of entrainment in spoken dialogue continues to pose a challenge. In this paper, we propose a straightforward definition for local entrainment in the speech domain and operationalize an algorithm based on this: acoustic-prosodic features that capture entrainment should be maximally different between real conversations involving two partners and sham conversations generated by randomly mixing the speaking turns from the original two conversational partners. We propose an approach for measuring local entrainment that quantifies alignment of behavior on a turn-by-turn basis, projecting the differences between interlocutors' acoustic-prosodic features for a given turn onto a discriminative feature subspace that maximizes the difference between real and sham conversations. We evaluate the method using the derived features to drive a classifier aiming to predict an objective measure of conversational success (i.e., low versus high), on a corpus of task-oriented conversations. The proposed entrainment approach achieves 72% classification accuracy using a Naive Bayes classifier, outperforming three previously established approaches evaluated on the same conversational corpus.

Classifying conversational entrainment of speech behavior: An expanded framework and review

“…Within this definition, measures of local entrainment are relatively straightforward. Speech features are generally compared between adjacent turns or adjacent IPUs (e.g., Schweitzer, Walsh, & Schweitzer, 2017;Willi et al, 2018).…”

Sync Pending: Characterizing Conversational Entrainment in Dysarthria Using a Multidimensional, Clinically Informed Approach

“…The measure of communicative efficiency is, therefore, an objective measure of how proficiently the dyad used verbal communication to collaboratively work through the demands of the goal-oriented dialogue task. This objective measure is used to examine the predictive relationship between acoustic entrainment and conversational success-a relationship previously reported with other acoustic measures of entrainment (e.g., Borrie et al, 2019Willi, Borrie, Barrett, Tu, & Berisha, 2018).…”

“…Five large acoustic feature sets were extracted from each individual spoken utterance, including (a) envelope modulation spectrum (EMS; 60 features), (b) rhythm metrics (12 features), (c) long-term average spectrum (LTAS; 99 features), (d) mel-frequency cepstral coefficients (MFCCs; 234 features), and (e) voice report (24 features). These feature sets, which have been previously reported on (e.g., Berisha, Liss, Sandoval, Utianski, & Spanias, 2014;Tu, Berisha, & Liss, 2017;Tu, Jiao, Berisha, & Liss, 2016;Willi et al, 2018), are considered to reflect rhythmic (EMS, rhythm metrics), articulatory (LTAS, MFCC), and phonatory (voice report) dimensions of speech signal behavior (e.g., Cleveland, Sunberg, & Stone, 2001;Dellwo, Fourcin, & Abberton, 2013;Liss, LeGendre, & Lotto, 2010; also see Borrie et al, 2019, for dimension justification). For comprehensive details of feature calculation, please refer to Supplemental Material S1.…”

Classifying Conversational Entrainment of Speech Behavior: An Expanded Framework and Review

“…Within this definition, measures of local entrainment are relatively straightforward. Acousticprosodic features are generally compared between adjacent turns or adjacent IPUs (e.g., Schweitzer, Walsh, & Schweitzer, 2017;Willi et al, 2018).…”

“…Our review of the literature provides evidence that all of these types have been used to capture entrainment (see also Table 1). The use 2014t-test comparing differences between feature values of conversation partners on adjacent and non-adjacent turns Willi et al (2018) Linear discriminant analysis using turn-level acoustic feature values Reichel et al (2018) Linear mixed models comparing the absolute difference between feature values of conversation partners in adjacent and non-adjacent turns…”