It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models

Abstract

When a NLU model is updated, new utterances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a production environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human annotation. Experimental results show that the proposed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1score based metric, while reducing the annotation volume by 97% and overall cost by 60% for each iteration.

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations

“…For instance, Promptsource (Bach et al, 2022), is a framework designed to try out diverse set of prompts that can be used in in-context learning , or instruction tuning (Sanh et al, 2021;Min et al, 2021;Ye et al, 2022;Jang et al, 2023). Other human-in-the-loop annotation toolkits Weber et al, 2021; provides functionality for annotators to verify the neural model's prediction instead of manually creating them. Compared to these toolkits, CoTEver 1 Example from StrategyQA (Geva et al, 2021b) provides additional features specifically designed for gathering explanation data such as retrieving evidence documents and supporting different Chain of Thought prompts.…”

“…In this paper, we address the question: can we gather explanation data in a more efficient manner? Inspired by human-in-the-loop methods, we ask annotators to verify a machine generated explanation instead of manually writing them Weber et al, 2021;. In other words, annotators get to check whether the underlying language model hallucinate (i.e., generate explanations that are factually incorrect) (Shuster et al, 2021;Lin et al, 2022a).…”

CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification

“…For instance, Promptsource (Bach et al, 2022), is a framework designed to try out diverse set of prompts that can be used in in-context learning , or instruction tuning (Sanh et al, 2021;Wei et al, 2021;Min et al, 2021;Ye et al, 2022;Jang et al, 2023). Other human-in-the-loop annotation toolkits (Wallace et al, 2019;Weber et al, 2021; provides functionality for annotators to verify the neural model's prediction instead of manually creating them. Compared to these toolkits, CoTEver 1 Example from StrategyQA (Geva et al, 2021b) provides additional features specifically designed for gathering explanation data such as retrieving evidence documents and supporting different Chain of Thought prompts.…”

“…In this paper, we address the question: can we gather explanation data in a more efficient manner? Inspired by human-in-the-loop methods, we ask annotators to verify a machine generated explanation instead of manually writing them (Wallace et al, 2019;Weber et al, 2021;. In other words, annotators get to check whether the underlying language model hallucinate (i.e., generate explanations that are factually incorrect) (Shuster et al, 2021;Lin et al, 2022a).…”

Designing a Human-in-the-Loop System for Object Detection in Floor Plans

“…For example, in the medical domain, it is essential that machine learning models forward x-ray images that are difficult to classify (i.e., the model is uncertain about the prediction) to physicians for manual inspection. In the literature, several setups exist in which human experts augment and complement ML models: HITL systems are employed in supervised learning (e.g., Wang et al 2016, Kamar 2016, Wu, Xiao, Sun, Zhang, Ma & He 2021, semi-supervised learning (e.g., Wrede & Hellander 2019, Weber et al 2021, and reinforcement learning (e.g., Wu et al 2022, Liang et al 2017, Elmalaki 2021). However, these approaches generally require repetitive human effort that is growing with the number of unknown instances and the inaccuracy in detecting such instances.…”