Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders

Zou, Yicheng; Jun, Lin; Zhao, Lujun; Kang, Yangyang; Jiang, Zhuoren; Sun, Chengjie; Zhang, Qi; Huang, Xuanjing; Liu, Xiaozhong

doi:10.1609/aaai.v35i16.17724

Cited by 9 publications

(2 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To structure chat content analyses and adjust the granularity of topics identified, future web-based symposia may consider incorporating interactive prompts throughout presentations in the chat to engage participants in specific thematic areas. Computationally, topic-oriented ranking with context-aware autoencoders, such as Bidirectional Encoder Representations from Transformers (BERT) may be an approach to improve topic model analyses of documents (eg, chat logs) with rapidly evolving, fragmented topics [ 48 ]. Finally, knowledge of participant demographics (eg, employment, specialty areas, age) may facilitate high-resolution network analyses of participant interactions and their level of importance [ 49 ].…”

Section: Discussionmentioning

confidence: 99%

Eliciting Insights From Chat Logs of the 25X5 Symposium to Reduce Documentation Burden: Novel Application of Topic Modeling

Moy¹,

Withall²,

Hobensack³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

Background Addressing clinician documentation burden through “targeted solutions” is a growing priority for many organizations ranging from government and academia to industry. Between January and February 2021, the 25 by 5: Symposium to Reduce Documentation Burden on US Clinicians by 75% (25X5 Symposium) convened across 2 weekly 2-hour sessions among experts and stakeholders to generate actionable goals for reducing clinician documentation over the next 5 years. Throughout this web-based symposium, we passively collected attendees’ contributions to a chat functionality—with their knowledge that the content would be deidentified and made publicly available. This presented a novel opportunity to synthesize and understand participants’ perceptions and interests from chat messages. We performed a content analysis of 25X5 Symposium chat logs to identify themes about reducing clinician documentation burden. Objective The objective of this study was to explore unstructured chat log content from the web-based 25X5 Symposium to elicit latent insights on clinician documentation burden among clinicians, health care leaders, and other stakeholders using topic modeling. Methods Across the 6 sessions, we captured 1787 messages among 167 unique chat participants cumulatively; 14 were private messages not included in the analysis. We implemented a latent Dirichlet allocation (LDA) topic model on the aggregated dataset to identify clinician documentation burden topics mentioned in the chat logs. Coherence scores and manual examination informed optimal model selection. Next, 5 domain experts independently and qualitatively assigned descriptive labels to model-identified topics and classified them into higher-level categories, which were finalized through a panel consensus. Results We uncovered ten topics using the LDA model: (1) determining data and documentation needs (422/1773, 23.8%); (2) collectively reassessing documentation requirements in electronic health records (EHRs) (252/1773, 14.2%); (3) focusing documentation on patient narrative (162/1773, 9.1%); (4) documentation that adds value (147/1773, 8.3%); (5) regulatory impact on clinician burden (142/1773, 8%); (6) improved EHR user interface and design (128/1773, 7.2%); (7) addressing poor usability (122/1773, 6.9%); (8) sharing 25X5 Symposium resources (122/1773, 6.9%); (9) capturing data related to clinician practice (113/1773, 6.4%); and (10) the role of quality measures and technology in burnout (110/1773, 6.2%). Among these 10 topics, 5 high-level categories emerged: consensus building (821/1773, 46.3%), burden sources (365/1773, 20.6%), EHR design (250/1773, 14.1%), patient-centered care (162/1773, 9.1%), and symposium comments (122/1773, 6.9%). Conclusions We conducted a topic modeling analysis on 25X5 Symposium multiparticipant chat logs to explore the feasibility of this novel application and elicit additional insights on clinician documentation burden among attendees. Based on the results of our LDA analysis, consensus building, burden sources, EHR design, and patient-centered care may be important themes to consider when addressing clinician documentation burden. Our findings demonstrate the value of topic modeling in discovering topics associated with clinician documentation burden using unstructured textual content. Topic modeling may be a suitable approach to examine latent themes presented in web-based symposium chat logs.

show abstract

Section: Discussionmentioning

confidence: 99%

Eliciting Insights From Chat Logs of the 25X5 Symposium to Reduce Documentation Burden: Novel Application of Topic Modeling

Moy¹,

Withall²,

Hobensack³

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…Unfortunately, standard supervised training of these models rely on large labeled datasets, which can be prohibitively expensive to build. While unsupervised summarization techniques exist (Zou et al, 2021;Shang et al, 2018), enforcing the summary quality and style is still an ongoing challenge.…”

Section: Introductionmentioning

confidence: 99%

LLM aided semi-supervision for efficient Extractive Dialog Summarization

Mishra,

Sahu,

Calixto

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Generating high-quality summaries for chat dialogs often requires large labeled datasets. We propose a method to efficiently use unlabeled data for extractive summarization of customeragent dialogs. In our method, we frame summarization as a question-answering problem and use state-of-the-art large language models (LLMs) to generate pseudo-labels for a dialog. We then use these pseudo-labels to finetune a chat summarization model, effectively transferring knowledge from the large LLM into a smaller specialized model. We demonstrate our method on the TWEETSUMM dataset, and show that using 10% of the original labelled data set we can achieve 65.9/57.0/61.0 ROUGE-1/-2/-L, whereas the current state-ofthe-art trained on the entire training data set obtains 65.16/55.81/64.37 ROUGE-1/-2/-L. In other words, in the worst case (i.e., ROUGE-L) we still effectively retain 94.7% of the performance while using only 10% of the data.

show abstract