Detecting conversation topics in primary care office visits from transcripts of patient-provider interactions

Abstract

Objective Amid electronic health records, laboratory tests, and other technology, office-based patient and provider communication is still the heart of primary medical care. Patients typically present multiple complaints, requiring physicians to decide how to balance competing demands. How this time is allocated has implications for patient satisfaction, payments, and quality of care. We investigate the effectiveness of machine learning methods for automated annotation of medical topics in patient-provider dialog transcripts. Materials and Methods We used dialog transcripts from 279 primary care visits to predict talk-turn topic labels. Different machine learning models were trained to operate on single or multiple local talk-turns (logistic classifiers, support vector machines, gated recurrent units) as well as sequential models that integrate information across talk-turn sequences (conditional random fields, hidden Markov models, and hierarchical gated recurrent units). Results Evaluation was performed using cross-validation to measure 1) classification accuracy for talk-turns and 2) precision, recall, and F1 scores at the visit level. Experimental results showed that sequential models had higher classification accuracy at the talk-turn level and higher precision at the visit level. Independent models had higher recall scores at the visit level compared with sequential models. Conclusions Incorporating sequential information across talk-turns improves the accuracy of topic prediction in patient-provider dialog by smoothing out noisy information from talk-turns. Although the results are promising, more advanced prediction techniques and larger labeled datasets will likely be required to achieve prediction performance appropriate for real-world clinical applications.

A Systematic Review of Machine Learning for Assessment and Feedback of Treatment Fidelity

“…Pérez-Rosas et al, 2019 showed that as they increased the amount of data in their training set they observed significant improvement in prediction accuracy. Aligned with this finding, frequently observed codes (i.e., categories) in a dataset Howes et al, 2013;Perez-Rosas et al, 2017;Perez-Rosas et al, 2019;Xiao et al, 2015;Park et al, 2019;Flemotomos et al, 2018. Random Forest 7 Carcone et al, 2019Imel et al, 2015;Mieskes and Stiegelmayr, 2018;Blanchard et al, 2016a;Blanchard et al, 2016b;Donnelly et al, 2017;Wang et al, 2014.…”

“…In this context, the machine learning methods were designed to automatically assign codes from the behavioural coding measures to overt interactions recorded in the dataset (e.g., words/utteran- (Atkins et al, 2014;Can et al, 2015;Can et al, 2012;Can et al, 2016;Cao et al, 2020;Carcone et al, 2019, Study 1;Chakravarthula et al, 2015;Chen et al, 2019;Gibson et al, 2017;Gibson et al, 2016;Gupta et al, 2014;Hasan et al, 2019;Hasan et al, 2018;Imel et al, 2015;Perez-Rosas et al, 2017;Perez-Rosas et al, 2019;Singla et al, 2018;Tanana et al, 2016;Xiao et al, 2012;Xiao et al, 2015;Xiao, Can, et al, 2016;Xiao, Huang, et al, 2016) Medical care, provider-patient clinical interactions (Carcone et al, 2019, Study 2;Park et al, 2019) Education (teachers) (Blanchard et al, 2016a;Blanchard et al, 2016b;Donelly et al, 2017;Donnely et al, 2016a;Donnelly et al, 2016b;Samei et al, 2014;Samei et al, 2015;Song et al, 2020;Suresh et al, 2019;Wang et al, 2014) Counselling, (counsellors), (Althoff et al, 2016;Flemotomos et al, 2018;Gallo et al, 2015;Gaut et al, 2017;Goldberg et al, 2020…”

“…Hidden Markov Model 4 Althoff et al, 2016;Can et al, 2012;Hasan et al, 2019;Hasan et al, 2018K-Nearest Neighbours 4 Blanchard et al, 2016aSen et al, 2017;Blanchard et al, 2016b;Donnelly et al, 2017 Conditional Random Field 4 Can et al, 2015;Carcone et al, 2019;Wallace et al, 2014;Park et al, 2019 Bi-directional Long Short Term Memory (Bi-LSTM) 3 Chen et al, 2019;Suresh et al, 2019 Labelled Topic Model 2 Atkins et al, 2014;Imel et al, 2015 Bayesian Network 2 Blanchard et al, 2016a;Blanchard et al, 2016b Gared Recurrent Unit (GRU) 2 Cao et al, 2019;Park et al, 2019 30 models were used once each 2 1 each (30 in total) Angus et al, 2012;Carcone et al, 2019;Chakravarthula et al, 2015;Gallo et al, 2015;Gaut et al, 2017;Gibson et al, 2016;Gibson et al, 2017;Hasan et al, 2018;Howes et al, 2013;Imel et al, 2015;Lacson et al, 2005;Malandrakis and Narayanan, 2015;Nitti et al, 2010;Salvatore et al, 2012;Tanana et al, 2016;Velasquez & Montiel, 2018;Wallace et al, 2013;Xiao et al, 2012;Xiao, Huang et al, 2016;Xiao, Can, et al, 2016;…”

Challenges of developing a digital scribe to reduce clinical documentation burden

“…One solution is to identify the category of each speaker turn (utterance), allowing for topic blocks to be identified in the transcripts (topic segmentation). 22,26 Targeted information extraction and summarization can then be applied to the identified topics. 7,22 The topics can be based on predetermined categories 22 or the components of a traditional medical encounter (chief complaint, family history, social history).…”

Utilization of Time Series Tools in Life-sciences and Neuroscience

“…HMM was used in detecting topics from transcripts of patient-provider conversations. 103 This could improve patient satisfaction, cost-effectiveness, and care. Infinite HMMs can be used as a powerful tool to analyze single molecule sequential data without any prior setting of stat, which is required infinite HMMs.…”