Investigating the relationships between dialog patterns and user satisfaction in customer service chat systems based on chat log analysis

Abstract

While previous studies of customer service chat systems (CSCS) understood user satisfaction as individuals’ subjective perceptions and depended heavily on self-report methods for satisfaction measurement, this article presents an obtrusive chat log analysis that followed the established approaches of search log analysis and examined the relationships between dialog patterns and user satisfaction. An 81-day chat log was obtained from a real-world CSCS that involves both a chatbot and human representatives. A total of 75,918 chat sessions/147,972 sub-sessions containing 251,556 user messages and 349,416 system messages were extracted after data processing and analysed in terms of topic, length and path. As found in this study, the users were more likely to get satisfied on low-difficulty topics. The dialog between the CSCS and users was shallow in general. While human representatives’ elaboration contributed to user satisfaction, the chatbot was responsible for damaging user satisfaction. The significance of this study consists not only in providing objective evidence about user satisfaction in online chat but also in generating practical implications for the CSCS to improve user satisfaction.