Enhancement of Conversational Agents By Means of Multimodal Interaction

Abstract

The main objective of multimodal conversational agents is to provide a more engaged and participative communication by allowing users to employ more than one input methodologies and providing output channels that are different to exclusively using voice. This chapter presents a detailed study on the benefits, disadvantages, and implications of incorporating multimodal interaction in conversational agents. Initially, it focuses on implementation techniques. Next, it explains the fusion and fission of multimodal information and focuses on the core module of these agents: the dialogue manager. Later on, the chapter addresses architectures, tools to develop some typical components of the agents, and evaluation methodologies. As a case of study, it describes the multimodal conversational agent in which we are working at the moment to provide assistance to professors and students in some of their daily activities in an academic centre, for example, a University’s Faculty.

Conversational agents in healthcare: a systematic review

“…Frame-based systems address some of the limitations of finite-state dialogue management, by allowing for system and mixed initiative, as well as enabling a more flexible dialogue 8 , 36 . Both methodologies are able to manage tasks based on the filling of a form by requesting data from the user.…”

“…We only identified one study that evaluated this type of conversational agent in the health context, and the agents were not designed specifically to answer health-related questions 26 . One of the major disadvantages of these systems—the fact that they require large training datasets—may be a reason for their slow adoption in health applications 36 , 42 …”

An overview of the features of chatbots in mental health: A scoping review

“…Further, they mainly use machine and deep learning to learn from gathered data and continuously improve their performance and responses [42,43]. Artificial intelligence chatbots also need large health databases and corpora in order for them to be trained [50]. Given the advancements in machine and deep learning and natural language processing and growing availability of large health databases and corpora [51,52], developers should endeavour to build more artificial intelligence chatbots.…”

Measuring User Experience in Conversational Interfaces: A Comparison of Six Questionnaires

“…A conversational system comprises various modules including Automatic Speech Recognition, Natural Language Understanding, Dialogue Management, Natural Language Generation, and Speech Synthesis (López-Cózar et al 2011). There are specific measures defined for each module.…”

“…There are specific measures defined for each module. For example, the word error rate is used for Automatic Speech Recognition, and the rate of out of vocabulary words is used for Natural Language Generation (López-Cózar et al 2011). In terms of UX, there is no specific module directly responsible for UX; rather, all system modules play a role in shaping UX.…”