A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering

Abdellatif, Ahmad; Badran, Khaled; Costa, Diego Elias; Shihab, Emad

doi:10.1109/tse.2021.3078384

Cited by 59 publications

(37 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These components provide both NLU and dialogue management services. Abdellatif et al (2021) evaluated the NLU components that are suitable for software engineering tasks. The comparison resulted in IBM Watson being ranked as the best for intent classification and entity extraction, whereas Rasa ranked the best for confidence scores.…”

Section: Resultsmentioning

confidence: 99%

Bots in software engineering: a systematic mapping study

Santhanam

Hecking

Schreiber

et al. 2022

PeerJ Computer Science

View full text Add to dashboard Cite

Bots have emerged from research prototypes to deployable systems due to the recent developments in machine learning, natural language processing and understanding techniques. In software engineering, bots range from simple automated scripts to decision-making autonomous systems. The spectrum of applications of bots in software engineering is so wide and diverse, that a comprehensive overview and categorization of such bots is needed. Existing works considered selective bots to be analyzed and failed to provide the overall picture. Hence it is significant to categorize bots in software engineering through analyzing why, what and how the bots are applied in software engineering. We approach the problem with a systematic mapping study based on the research articles published in this topic. This study focuses on classification of bots used in software engineering, the various dimensions of the characteristics, the more frequently researched area, potential research spaces to be explored and the perception of bots in the developer community. This study aims to provide an introduction and a broad overview of bots used in software engineering. Discussions of the feedback and results from several studies provide interesting insights and prospective future directions.

show abstract

Section: Resultsmentioning

confidence: 99%

Bots in software engineering: a systematic mapping study

Santhanam

Hecking

Schreiber

et al. 2022

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Also, we believe that there is a need for more studies that compare different NLUs using more datasets to benchmark NLUs in the SE context. We contribute towards this effort by making our dataset publicly available [31]. P: Precision, R: Recall, F1: F1-measure…”

Section: Discussionmentioning

confidence: 99%

“…After the merge, we discarded queries with unclear intent (total of 82), such as "JConsole Web Application". The final set includes 215 queries [31], Tables 4 and 3 show the number of intents and entities included in our evaluation, respectively.…”

Section: Facingerrormentioning

confidence: 99%

See 1 more Smart Citation

A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering

Abdellatif,

Badran,

Costa

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Chatbots are envisioned to dramatically change the future of Software Engineering, allowing practitioners to chat and inquire about their software projects and interact with different services using natural language. At the heart of every chatbot is a Natural Language Understanding (NLU) component that enables the chatbot to understand natural language input. Recently, many NLU platforms were provided to serve as an off-the-shelf NLU component for chatbots, however, selecting the best NLU for Software Engineering chatbots remains an open challenge. Therefore, in this paper, we evaluate four of the most commonly used NLUs, namely IBM Watson, Google Dialogflow, Rasa, and Microsoft LUIS to shed light on which NLU should be used in Software Engineering based chatbots. Specifically, we examine the NLUs' performance in classifying intents, confidence scores stability, and extracting entities. To evaluate the NLUs, we use two datasets that reflect two common tasks performed by Software Engineering practitioners, 1) the task of chatting with the chatbot to ask questions about software repositories 2) the task of asking development questions on Q&A forums (e.g., Stack Overflow). According to our findings, IBM Watson is the best performing NLU when considering the three aspects (intents classification, confidence scores, and entity extraction). However, the results from each individual aspect show that, in intents classification, IBM Watson performs the best with an F1-measure>84%, but in confidence scores, Rasa comes on top with a median confidence score higher than 0.91. Our results also show that all NLUs, except for Dialogflow, generally provide trustable confidence scores. For entity extraction, Microsoft LUIS and IBM Watson outperform other NLUs in the two SE tasks. Our results provide guidance to software engineering practitioners when deciding which NLU to use in their chatbots.

show abstract

“…In the first version of our Weaver platform, we chose Dialogflow as the NLU service. When we saw some shortcomings, we introduced RASA as the most trustworthy opensource NLU service in terms of confidence score [32]. After that, we realized the problem of NLU dependency and then came up with the idea we have implemented in our architecture.…”

Section: Nlu Supportmentioning

confidence: 99%

Extensible Chatbot Architecture Using Metamodels of Natural Language Understanding

et al. 2021

View full text Add to dashboard Cite

In recent years, gradual improvements in communication and connectivity technologies have enabled new technical possibilities for the adoption of chatbots across diverse sectors such as customer services, trade, and marketing. The chatbot is a platform that uses natural language processing, a subset of artificial intelligence, to find the right answer to all users’ questions and solve their problems. Advanced chatbot architecture that is extensible, scalable, and supports different services for natural language understanding (NLU) and communication channels for interactions of users has been proposed. The paper describes overall chatbot architecture and provides corresponding metamodels as well as rules for mapping between the proposed and two commonly used NLU metamodels. The proposed architecture could be easily extended with new NLU services and communication channels. Finally, two implementations of the proposed chatbot architecture are briefly demonstrated in the case study of “ADA” and “COVID-19 Info Serbia”.

show abstract

A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering

Cited by 59 publications

References 42 publications

Bots in software engineering: a systematic mapping study

Bots in software engineering: a systematic mapping study

A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering

Extensible Chatbot Architecture Using Metamodels of Natural Language Understanding

Contact Info

Product

Resources

About