“It sounds like...”: A natural language processing approach to detecting counselor reflections in motivational interviewing.

Abstract

The dissemination and evaluation of evidence based behavioral treatments for substance abuse problems rely on the evaluation of counselor interventions. In Motivational Interviewing (MI), a treatment that directs the therapist to utilize a particular linguistic style, proficiency is assessed via behavioral coding - a time consuming, non-technological approach. Natural language processing techniques have the potential to scale up the evaluation of behavioral treatments like MI. We present a novel computational approach to assessing components of MI, focusing on one specific counselor behavior – reflections – that are believed to be a critical MI ingredient. Using 57 sessions from 3 MI clinical trials, we automatically detected counselor reflections in a Maximum Entropy Markov Modeling framework using the raw linguistic data derived from session transcripts. We achieved 93% recall, 90% specificity, and 73% precision. Results provide insight into the linguistic information used by coders to make ratings and demonstrate the feasibility of new computational approaches to scaling up the evaluation of behavioral treatments.

A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing

“…The two methods, introduced and described below, differ substantially in terms of their complexity and their conceptualization of language and linguistic structure. Moreover, they are fundamentally different models relative to topic models or maximum entropy models tested in previous research (Atkins et al, 2014; Can et al, in press; Imel et al, 2015). The most important difference between the models tested in this study and topic models is that these models attempt to incorporate the linguistic structure of a sentence beyond the mere presence of words and phrases.…”

“…In one of the first tests of computer-predicted coding in MI, Can, Marin, Georgiou, Imel, Atkins, and Narayanan (in press) used maximum entropy Markov models to predict clinician reflections based on the occurrence of words and phrases within utterances (e.g., “it sounds like”) as well as the amount of overlap of words and phrases between clinician and client speech. They obtained good accuracy for predicting reflections (positive predictive value = 0.73, sensitivity = 0.93), however, this model only predicted clinician reflections but not other types of clinician or client speech and was limited to 57 sessions.…”

Transitional Dynamics of Three Supervisory Styles Using Markov Chain Analysis

“…Furthermore, this analysis is not limited to studying supervision dynamics. Duys and Headrick (2004) used Markov chain analysis to examine hypothesized differences in using counseling skills between students who were rated as “effective” and those rated as “ineffective.” Can et al (2016) used the maximum entropy Markov model to detect counselor reflections in motivational interviewing, which scaled up the evaluation of behavioral treatments.…”

"It's hard to argue with a computer"

“…The ML / NLP system in CORE-MI was trained on a dataset of 300K utterances from 356 MI session recordings, which were hand labeled by an 8-person coding team using established MISC coding protocols [31]. Various machine learning models have been evaluated for predicting MISC codes (i.e., MI tone) features [9][45][51]. Averaging over all counselor codes, the correlation of model-based predictions with human-generated codes was found to be 93% of human reliability (SD = 16%) [7].…”