ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents

Abstract

We present ADVISER 1 -an open-source, multi-domain dialog system toolkit that enables the development of multi-modal (incorporating speech, text and vision), sociallyengaged (e.g. emotion recognition, engagement level prediction and backchanneling) conversational agents. The final Python-based implementation of our toolkit is flexible, easy to use, and easy to extend not only for technically experienced users, such as machine learning researchers, but also for less technically experienced users, such as linguists or cognitive scientists, thereby providing a flexible platform for collaborative research.

Emerging Technologies of Natural Language‐Enabled Chatbots: A Review and Trend Forecast Using Intelligent Ontology Extraction and Patent Analytics

“…e early development of chatbot was mostly dominated by a single domain. It has been observed that more research has been directed towards open domain [45][46][47][48][49] and multidomain [50][51][52] in recent years. Single-domain chatbots are limited to accomplishing specific tasks, while multidomain or open domain chatbots can better meet the needs of smart assistants and even further provide people's companionship or social applications.…”

Personalized and personal conversations : designing agents who want to know you

“…Designing multimodal systems is becoming increasingly more feasible than ever with many tools and frameworks being developed. Most of them are online available via public repositories (Bohus et al, 2017;Michael, 2020;Li et al, 2020). However, many dialogue designers still struggle with the complexity of building social conversational agents.…”

“…An editor for modifying template files in a tree-like structure would benefit less technically apt dialogue designers. Nevertheless, we see Flipper as a more abstract dialogue management system, connected to external components that each have their own authoring interfaces that help dialogue designers in prototyping a specific component, such as the user simulations in ADvISER (Li et al, 2020).…”

“…In the VHToolkit (Hartholt et al, 2013), WAMI (Gruenstein et al, 2008) and Flipper scripting is also possible for less restricted authoring. ADvISER (Li et al, 2020) and ConvLab-2 (Zhu et al, 2020) support user simulations via a GUI to evaluate authored dialogues and diagnose possible issues.…”

“…ADvISER is a dialogue tool that focuses on helping people with less affinity with development such as linguists and cognitive scientists to design dialogues. The tool features multimodal processing pipelines, such as handlers for the latest ASR technology and OpenFace for recognizing faces (Li et al, 2020). However, the tool does not have a user-friendly interface and still requires some coding for setting up and customizing to a domain.…”