Topic Modeling as a Tool for Analyzing Library Chat Transcripts

Abstract

Library chat services are an increasingly important communication channel to connect patrons to library resources and services. Analysis of chat transcripts could provide librarians with insights into improving services. Unfortunately, chat transcripts consist of unstructured text data, making it impractical for librarians to go beyond simple quantitative analysis (e.g., chat duration, message count, word frequencies) with existing tools. As a stepping-stone toward a more sophisticated chat transcript analysis tool, this study investigated the application of different types of topic modeling techniques to analyze one academic library’s chat reference data collected from April 10, 2015, to May 31, 2019, with the goal of extracting the most accurate and easily interpretable topics. In this study, topic accuracy and interpretability—the quality of topic outcomes—were quantitatively measured with topic coherence metrics. Additionally, qualitative accuracy and interpretability were measured by the librarian author of this paper depending on the subjective judgment on whether topics are aligned with frequently asked questions or easily inferable themes in academic library contexts. This study found that from a human’s qualitative evaluation, Probabilistic Latent Semantic Analysis (pLSA) produced more accurate and interpretable topics, which is not necessarily aligned with the findings of the quantitative evaluation with all three types of topic coherence metrics. Interestingly, the commonly used technique Latent Dirichlet Allocation (LDA) did not necessarily perform better than pLSA. Also, semi-supervised techniques with human-curated anchor words of Correlation Explanation (CorEx) or guided LDA (GuidedLDA) did not necessarily perform better than an unsupervised technique of Dirichlet Multinomial Mixture (DMM). Last, the study found that using the entire transcript, including both sides of the interaction between the library patron and the librarian, performed better than using only the initial question asked by the library patron across different techniques in increasing the quality of topic outcomes.

Identify Reviews of Pedulilindungi Applications using Topic Modeling with Latent Dirichlet Allocation Method

“…A set of words generated on a model topic with a value based on the level of coherence or in human translation or interpretation with each level of ease is referred to as Topic coherence. Having a measure of the value of a topic, Topic coherence estimates the status or rank of semantic equality between words in a case [12].…”

Exploring an automated method for the analysis of virtual reference interactions

“…While using Python and regular expression are starting to be used in the evaluation of reference interactions, they have been employed as a research method in the humanities for a number of years (Benito-Santos and Sánchez, 2020; Koh and Fienup, 2021; Ozeran and Martin, 2019; Sharma et al. , 2022).…”

Using Machine Learning and Natural Language Processing to Analyze Library Chat Reference Transcripts

“…Their study applied pLSA (Probabilistic Latent Semantic Analysis) to library chat data over a period of four years, resulting in more accurate and interpretable topics and subjects compared with results by human qualitative evaluation. 8 Another interesting ML project on chat reference data was conducted by Ellie Kohler. This project used a machine learning model to analyze chat transcripts for sentiment and topic extraction.…”