LIDA: Lightweight Interactive Dialogue Annotator

Abstract

Dialogue systems have the potential to change how people interact with machines but are highly dependent on the quality of the data used to train them. It is therefore important to develop good dialogue annotation tools which can improve the speed and quality of dialogue data annotation. With this in mind, we introduce LIDA, an annotation tool designed specifically for conversation data. As far as we know, LIDA is the first dialogue annotation system that handles the entire dialogue annotation pipeline from raw text, as may be the output of transcription services, to structured conversation data. Furthermore it supports the integration of arbitrary machine learning models as annotation recommenders and also has a dedicated interface to resolve inter-annotator disagreements such as after crowdsourcing annotations for a dataset. LIDA is fully open source, documented and publicly available 1 .

Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research

“…Previous work required a minimum of 100 positive examples for each code [47], while participants in our evaluation, on average, only created 133 (MAXQDA) or 182 (Cody) positive examples overall. Our participant Kelly reported the most interaction with ML suggestions 9 , while others barely noticed them. We believe that the barriers we set for Cody to providing ML suggestions, namely defning cut-of values for prediction confdence and requiring labels to be predicted correctly for all test instances, helped flter out many wrong suggestions.…”

Survey on evaluation methods for dialogue systems

“…Another dialogue annotation tool is called LIDA (Collins et al 2019). The authors argue that the quality of a dataset has a significant effect on the quality of a dialogue system, hence, a good dialogue annotation tool is essential to create the best annotated dialogue dataset.…”

Situation-Based Multiparticipant Chat Summarization: a Concept, an Exploration-Annotation Tool and an Example Collection

“…TWIST (Pluss, 2012) and LIDA (Collins et al, 2019) are intended for dialogue annotation, which has been mostly focused on task-oriented dialogue for dialogue systems. Task-oriented dialogues already suppose a predefined topic and predefined roles (e.g., customer support tasks) and little noise.…”