A Dynamic Model of Speech for the Social Sciences

Abstract

Speech and dialogue are the heart of politics: nearly every political institution in the world involves verbal communication. Yet vast literatures on political communication focus almost exclusively on what words were spoken, entirely ignoring how they were delivered—auditory cues that convey emotion, signal positions, and establish reputation. We develop a model that opens this information to principled statistical inquiry: the model of audio and speech structure (MASS). Our approach models political speech as a stochastic process shaped by fixed and time-varying covariates, including the history of the conversation itself. In an application to Supreme Court oral arguments, we demonstrate how vocal tone signals crucial information—skepticism of legal arguments—that is indecipherable to text models. Results show that justices do not use questioning to strategically manipulate their peers but rather engage sincerely with the presented arguments. Our easy-to-use R package, communication, implements the model and many more tools for audio analysis.

Advancing an Interdisciplinary Science of Conversation: Insights from a Large Multimodal Corpus of Human Speech

“…One untapped form of inquiry is what we refer to as intra-conversational forecasting, that is, how an intra-conversational feature such as a speaker's tone of voice might influence a listener's own reactions during that same turn or carried over into subsequent turns (consider how laughter can be infectious). The premise of intra-conversational forecasting-or the "flow of conversation" (Knox & Lucas, 2021)-is especially intriguing as a subject for multimodal analysis; for example, a sudden shift in a listener's facial expression (e.g., a frown or a fading smile) may prompt the speaker to change the topic or soften their tone. In an unpublished exploratory analysis, we found suggestive evidence that participants often fall into identifiable typologies of verbal and nonverbal "leading" and "following," such that one person consistently steered the semantic content or emotional tone of a conversation.…”

Revealing racial bias

“…W e a r e n o w d e v e l o p i n g a s y s t e m f o r reconstructing encounter timelines from body-worn camera footage, building on my prior work on conversation and vocal-tone analysis in audio data (21,22) and incorporating our collaborators' expertise in computer vision (23,24). We continue to develop new causal methods for enabling the systematic evaluation of officers and agencies by fusing existing, imperfect datasets (25).…”

“In” or “Out”? An Analysis of the Use of Augmentals in U.S. Presidential Speeches on the Paris Climate Agreement

“…Because augmentals in political speeches, including REAs and PEAs, are usually established by connecting long-term, remote consequences with current actions, it is not surprising to see that the relational frames primarily involved deictic, causal, and coordination relations. In addition, political speeches tend to focus on acclaiming the stances and virtues of the speaking party, attacking the opposition, and defending against oppositional attacks (Benoit et al, 1998(Benoit et al, , 2000Knox & Lucas, 2021). Thus, relational frames such as comparison, distinction, and coordination can be expected to be used frequently.…”