A Performance Comparison of Different Cloud-Based Natural Language Understanding Services for an Italian e-Learning Platform

Abstract

During the COVID-19 pandemic, the corporate online training sector has increased exponentially and online course providers had to implement innovative solutions to be more efficient and provide a satisfactory service. This paper considers a real case study in implementing a chatbot, which answers frequently asked questions from learners on an Italian e-learning platform that provides workplace safety courses to several business customers. Having to respond quickly to the increase in the courses activated, the company decided to develop a chatbot using a cloud-based service currently available on the market. These services are based on Natural Language Understanding (NLU) engines, which deal with identifying information such as entities and intentions from the sentences provided as input. To integrate a chatbot in an e-learning platform, we studied the performance of the intent recognition task of the major NLU platforms available on the market with an in-depth comparison, using an Italian dataset provided by the owner of the e-learning platform. We focused on intent recognition, carried out several experiments and evaluated performance in terms of F-score, error rate, response time, and robustness of all the services selected. The chatbot is currently in production, therefore we present a description of the system implemented and its results on the original users’ requests.

Productivity of incident management with conversational bots-a review

“…Similarly, according to, [39], [46], [61] there is a segment of users who use these bots in Processes for personal assistance for various reasons. It was also found in [20], [28], [32], [33], [36], [38], [58], [59], [63], that chatbots are used in customer service processes. In this analysis, the application of chatbots in incident management processes has not been identified; however, it has been mentioned that the teaching process (student-delivered course), customer service process (customer-supplier), health process (patient-doctor), processes for personal assistance (assisted-assistant) and even the training of bots (trained-trainer), are carried out by conversational bots; in the case of the Incident Management Process (user-technician), its application is highly necessary and important to increase productivity in this process.…”

“…[26], [37], [46]- [48], [53], [54], [59], [69] Customer service processes Customer management process.…”

“…According to [32]- [34], mentioned that these bots offer assistance in patient care, which is considered to have been and continues to be a very valuable aspect. Likewise, the new normal of isolation and health care generated that businesses offering their products and services, [55]- [63] can be benefited with ease of information dissemination and customer service, also according to what was mentioned by [47] helps to understand customers; and according to [59]. The benefit of delegating tasks; whereas, in the case of [55]- [57].…”

“…In this analysis, the application of chatbots in incident management processes has not been identified; however, it has been mentioned that the teaching process (student-delivered course), customer service process (customer-supplier), health process (patient-doctor), processes for personal assistance (assisted-assistant) and even the training of bots (trained-trainer), are carried out by conversational bots; in the case of the Incident Management Process (user-technician), its application is highly necessary and important to increase productivity in this process. [37], [46]- [48], [53], [54], [59], [65] 3 Customer service processes 9 [23], [31], [35], [36], [39], [41], [61], [62], [66] 4 Health processes 11 [24], [25], [28], [29], [32]- [34], [43], [44], [56], [57] 5 Processes for personal assistance 3 [42], [49], […”

Assessing Performance of Cloud-Based Heterogeneous Chatbot Systems and A Case Study

“…• Execution Time • Startup Delay • Deadline Miss Rate • Incurred Cost Metaheuristics for scheduling of heterogeneous tasks in cloud • Make Span • Resource Utilization Cost computing environments: Analysis, performance evaluation, and future directions [31] Performance Evaluation of IoT Data Management Using MongoDB Versus MySQL Databases in Different Cloud Environments [41]Forkan et al[43] propose a tool for empirical evaluation, introducing metrics like Average Response time, Fallback Rate, Comprehensive Rate, Accuracy, Precision, Recall, and F1-score. Zubani et al[44] compare performance using metrics like F1-score, error rate, response time, and robustness. Carolyn F. Salazar[45] evaluates a Pedagogical Conversational Agent using a Cloud-Based Chatbot Builder, employing a black-box method and a checklist to assess adherence to international standards.…”