Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains

Schulz, Cláudia; Meyer, Christian M.; Kiesewetter, Jan; Sailer, Michael; Bauer, Elisabeth; Fischer, Martin R.; Fischer, Frank; Gurevych, Iryna

doi:10.18653/v1/p19-1265

Cited by 16 publications

(26 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Annotation suggestions. To evaluate the effects of providing annotation suggestions, we have conducted an extensive study (Schulz et al, 2019b) considering annotation time, annotation quality, potential biases, and the ease of use. To this end, we asked five Med and four TEd instructors to annotate diagnostic texts.…”

Section: Discussionmentioning

confidence: 99%

FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

Pfeiffer¹,

Meyer²,

Schulz³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

Our proposed system FAMULUS helps students learn to diagnose based on automatic feedback in virtual patient simulations, and it supports instructors in labeling training data.Diagnosing is an exceptionally difficult skill to obtain but vital for many different professions (e.g., medical doctors, teachers). Previous case simulation systems are limited to multiple-choice questions and thus cannot give constructive individualized feedback on a student's diagnostic reasoning process. Given initially only limited data, we leverage a (replaceable) NLP model to both support experts in their further data annotation with automatic suggestions, and we provide automatic feedback for students. We argue that because the central model consistently improves, our interactive approach encourages both students and instructors to recurrently use the tool, and thus accelerate the speed of data creation and annotation. We show results from two user studies on diagnostic reasoning in medicine and teacher education and outline how our system can be extended to further use cases.

show abstract

Section: Discussionmentioning

confidence: 99%

FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

Pfeiffer¹,

Meyer²,

Schulz³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

show abstract

“…Whereas existing work reports no measurable bias for expert annotators (Fort and Sagot, 2010;Lingren et al, 2014;Schulz et al, 2019), it remains unclear for annotators who have no prior experience in similar annotation tasks; especially for scenarios where -besides annotation guidelines -no further training is provided. However, the use of novice annotators is common for sce-0 1 10 100 1.000 10.000 2 0 1 9 -1 2 -0 1 2 0 2 0 -0 1 -0 1 2 0 2 0 -0 2 -0 1 2 0 2 0 -0 3 -0 1 2 0 2 0 -0 4 -0 1 Number of tweets narios where no linguistic or domain expertise is required.…”

Section: Related Workmentioning

confidence: 99%

“…The last group of students receives expert label suggestions in the first round and interactively updated label suggestions in the second round. In contrast to existing work (Schulz et al, 2019), this setup allows us to directly quantify effects of bias amplification that may occur with interactive label suggestions.…”

Section: Static Label Suggestions (G2)mentioning

confidence: 99%

Investigating label suggestions for opinion mining in German Covid-19 social media

Beck¹,

Lee²,

Viehmann³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement -in terms of inter-annotator agreement (+.14 Fleiss' κ) and annotation quality -compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups.To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations. 1

show abstract

“…Zhang et al (2020) investigate a human-in-the-loop approach for image segmentation and annotation. Schulz et al (2019) examine the use of suggestion models to support human experts with seg-mentation and classification of epistemic activities in diagnostic reasoning texts. Zhang and Chaudhuri (2015) suggest active learning from weak and strong labelers where these labelers can be humans with different levels of expertise in the labelling task.…”

Section: Related Workmentioning

confidence: 99%

It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models

Weber¹,

Piovano²,

Bradford³

2021

Proceedings of the Second Workshop on Data Science With Human in the Loop: Language Advances

View full text Add to dashboard Cite

When a NLU model is updated, new utterances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a production environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human annotation. Experimental results show that the proposed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1score based metric, while reducing the annotation volume by 97% and overall cost by 60% for each iteration.

show abstract

Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains

Cited by 16 publications

References 26 publications

FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

Investigating label suggestions for opinion mining in German Covid-19 social media

It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models

Contact Info

Product

Resources

About