ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems

Abstract

We present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of ConvLab (Lee et al., 2019b), ConvLab-2 inherits ConvLab's framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. The analysis tool presents rich statistics and summarizes common mistakes from simulated dialogues, which facilitates error analysis and system improvement. The interactive tool provides a user interface that allows developers to diagnose an assembled dialogue system by interacting with the system and modifying the output of each system component.

CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

“…To facilitate further research, we provide benchmark models for different components of a pipelined task-oriented dialogue system (Figure 3), including natural language understanding (NLU), dialogue state tracking (DST), dialogue policy learning, and natural language generation (NLG). These models are implemented using ConvLab-2 (Zhu et al, 2020), an open-source task-oriented dialog system toolkit. We also provide a rulebased user simulator, which can be used to train dialogue policy and generate simulated dialogue data.…”

Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking

“…For the Multilingual WoZ experiments, we followed the hyperparameters listed in Qin et al (2020) All of our hyperparameters for all the experiments will be made available as config files. We use code from Zhu et al (2020b) Dataset details: The dialogue state tracking datasets are available at the code repositories of Zhu et al (2020b) and Qin et al (2020) respectively. The OpenSubtitles corpus can be obtained from the corpus website 2 which is based on the subtitles website 3 .…”

Personalized and personal conversations : designing agents who want to know you

“…Commercial cloud-based services such as LUIS.AI, Wit.ai, DialogFlow, Watson, Lex and SAP Conversational AI all use neural networks for processing the information, 3 which is a useful approach for learning from large datasets containing text or conversations (Braun et al, 2017;Canonico and Russis, 2018). Similar open-source neural network approaches are RASA (Bocklisch et al, 2017) and ConvLab-2 (Zhu et al, 2020).…”

“…In the VHToolkit (Hartholt et al, 2013), WAMI (Gruenstein et al, 2008) and Flipper scripting is also possible for less restricted authoring. ADvISER (Li et al, 2020) and ConvLab-2 (Zhu et al, 2020) support user simulations via a GUI to evaluate authored dialogues and diagnose possible issues.…”

“…Developing dialogues can be done via a bottom-up pragmatic approach, a more theory-driven robust manner or a mixed approach, which are the design paradigms. design paradigms PyDial (Ultes et al, 2017), RASA (Bocklisch et al, 2017), ConvLab-2 (Zhu et al, 2020) and the commercial tools such as DialogFlow are more on the pragmatic side of the design paradigm scale, for quickly developing content with conversational data. Tools such as RavenClaw (Bohus and Rudnicky, 2009) and Disco (Rich and Sidner, 2012) require a theory-driven approach due to their hierarchical way of processing information and are not capable of using conversational data directly for development.…”