A Hybrid Neural Network BERT-Cap Based on Pre-Trained Language Model and Capsule Network for User Intent Classification

Abstract

User intent classification is a vital component of a question-answering system or a task-based dialogue system. In order to understand the goals of users’ questions or discourses, the system categorizes user text into a set of pre-defined user intent categories. User questions or discourses are usually short in length and lack sufficient context; thus, it is difficult to extract deep semantic information from these types of text and the accuracy of user intent classification may be affected. To better identify user intents, this paper proposes a BERT-Cap hybrid neural network model with focal loss for user intent classification to capture user intents in dialogue. The model uses multiple transformer encoder blocks to encode user utterances and initializes encoder parameters with a pre-trained BERT. Then, it extracts essential features using a capsule network with dynamic routing after utterances encoding. Experiment results on four publicly available datasets show that our model BERT-Cap achieves a F1 score of 0.967 and an accuracy of 0.967, outperforming a number of baseline methods, indicating its effectiveness in user intent classification.

Comparative Study of K-Nearest Neighbour and Naïve Bayes Performances on Malay Text Classification

“…Text classification (also known as text categorization) is a critical component of text analysis, which is the process of assigning appropriate predefined labels or tags to unstructured text such as phrases, paragraphs, questions, or documents in order to solve a variety of natural language processing problems. It is widely used in a wide range of applications, including spam email detection (Sharma et al, 2021;Ma, Yamamori & Thida, 2020;Taylor & Ezekiel, 2020), sentiment analysis (Waheeb et al, 2020;Alshamsi et al, 2020;Sudhir & Suresh, 2021), question answering (Perevalov & Both, 2021), news categorization (Fanny, Muliono & Tanzil, 2018;Mallick, Mishra & Chae, 2020), and user intent classification (Liu et al, 2020). Classification of pertinent text has demonstrated significant promise in a variety of domains, including marketing, product management, customer service, medical, and others.…”