Evaluating different Natural Language Understanding services in a real business case for the Italian language

Zubani, Matteo; Sigalini, Luca; Serina, Ivan; Gerevini, Alfonso

doi:10.1016/j.procs.2020.09.095

Cited by 10 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It presents the comparison of existing NLU services like DialogFlow (Google), Watson (IBM), and Alexa (Amazon). Study [94] focuses on the Italian language as a non-English language for using the NLU engine.…”

Section: Rq3 Which Are the Learning Methods For Improving The Learnin...mentioning

confidence: 99%

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

et al. 2022

View full text Add to dashboard Cite

Learning human languages is a difficult task for a computer. However, Deep Learning (DL) techniques have enhanced performance significantly for almost all-natural language processing (NLP) tasks. Unfortunately, these models cannot be generalized for all the NLP tasks with similar performance. NLU (Natural Language Understanding) is a subset of NLP including tasks, like machine translation, dialogue-based systems, natural language inference, text entailment, sentiment analysis, etc. The advancement in the field of NLU is the collective performance enhancement in all these tasks. Even though MTL (Multi-task Learning) was introduced before Deep Learning, it has gained significant attention in the past years. This paper aims to identify, investigate, and analyze various language models used in NLU and NLP to find directions for future research. The Systematic Literature Review (SLR) is prepared using the literature search guidelines proposed by Kitchenham and Charters on various language models between 2011 and 2021. This SLR points out that the unsupervised learning methodbased language models show potential performance improvement. However, they face the challenge of designing the generalpurpose framework for the language model, which will improve the performance of multi-task NLU and the generalized representation of knowledge. Combining these approaches may result in a more efficient and robust multi-task NLU. This SLR proposes building steps for a conceptual framework to achieve goals of enhancing the performance of language models in the field of NLU. INDEX TERMSDeep learning, Knowledge representation, Multi-task NLU, Unsupervised learningTABLE I APPLICATION DOMAINS FOR NLU Domain Applications Machine translation IBM Watson Task-based dialogue-based systems Booking tickets, taking an appointment using Google assistant

show abstract

Section: Rq3 Which Are the Learning Methods For Improving The Learnin...mentioning

confidence: 99%

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Generative models capable of automatically generating answers considering current and previous user messages are also in production, but there are still difficulties in building and training them and they are not available in the major commercial platforms (Adamopoulou and Moussiades 2020). Studies on the evaluation of these platforms show similar performances in terms of combined f-score (Liu et al 2019;Braun et al 2017), with slight differences in intent identification task (Canonico and De Russis 2018), especially with longer utterances (Zubania et al 2020).…”

Section: A Methodological Lack In Conversation Designmentioning

confidence: 99%

“…expand and enrich the procedural workflow with the implementation of a selection of patterns on one of the most authoritative NLU commercial platforms also for the Italian language, namely Google's Dialogflow (Zubania et al 2020).…”

Section: Wrongmentioning

confidence: 99%

Toward a linguistically grounded dialog model for chatbot design

Dall’Acqua¹,

Tamburini²

2021

ijcol

View full text Add to dashboard Cite

The increasing interest in various types of conversational interfaces has been supported by a progressive standardization of the technological frameworks used to build them. However, the landscape of available methodological frameworks for designing conversations is much more fragmented. We propose a highly generalizable methodology for designing conversational flows rooted in a functionalist-pragmatics perspective, with an explicit adherence to a conversationalist approach. In parallel, we elaborate a practical-procedural workflow for undertaking chatbots projects in which we situate the theoretical starting point. At last, we elaborate a general casestudy on which we transpose the identified approach in Italian language and using one of the most authoritative NLU platforms.

show abstract

“…Regarding the domain concerning this paper, the work in [48] presents a preliminary analysis of the performance of cloud-based NLU platforms. In this work, we substantially extend that analysis, introducing new results and several improvements, in particular focusing on evaluating the most important characteristics required for an implementation in the real world.…”

Section: Related Workmentioning

confidence: 99%

A Performance Comparison of Different Cloud-Based Natural Language Understanding Services for an Italian e-Learning Platform

Zubani

Sigalini²,

Serina

et al. 2022

Future Internet

Self Cite

View full text Add to dashboard Cite

During the COVID-19 pandemic, the corporate online training sector has increased exponentially and online course providers had to implement innovative solutions to be more efficient and provide a satisfactory service. This paper considers a real case study in implementing a chatbot, which answers frequently asked questions from learners on an Italian e-learning platform that provides workplace safety courses to several business customers. Having to respond quickly to the increase in the courses activated, the company decided to develop a chatbot using a cloud-based service currently available on the market. These services are based on Natural Language Understanding (NLU) engines, which deal with identifying information such as entities and intentions from the sentences provided as input. To integrate a chatbot in an e-learning platform, we studied the performance of the intent recognition task of the major NLU platforms available on the market with an in-depth comparison, using an Italian dataset provided by the owner of the e-learning platform. We focused on intent recognition, carried out several experiments and evaluated performance in terms of F-score, error rate, response time, and robustness of all the services selected. The chatbot is currently in production, therefore we present a description of the system implemented and its results on the original users’ requests.

show abstract

Evaluating different Natural Language Understanding services in a real business case for the Italian language

Cited by 10 publications

References 2 publications

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

Toward a linguistically grounded dialog model for chatbot design

A Performance Comparison of Different Cloud-Based Natural Language Understanding Services for an Italian e-Learning Platform

Contact Info

Product

Resources

About