Empowering Active Learning to Jointly Optimize System and User Demands

Lee, Ji-Ung; Meyer, Christian M.; Gurevych, Iryna

doi:10.18653/v1/2020.acl-main.390

Cited by 6 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adaptive estimators. While simple heuristics or annotator-unaware models allow us to pre-compute annotation curricula, they do not consider any user-specific aspect that may influence the difficulty estimation (Lee, Meyer, and Gurevych 2020). Consequently, the resulting curriculum may not provide the optimal ordering for a specific annotator.…”

Section: Masked Language Modeling Loss (Mlm)mentioning

confidence: 99%

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee

Klie

Gurevych

2022

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations; especially in citizen science or crowdsourcing scenarios where domain expertise is not required. To alleviate these issues, this work proposes annotation curricula, a novel approach to implicitly train annotators. The goal is to gradually introduce annotators into the task by ordering instances to be annotated according to a learning curriculum. To do so, this work formalizes annotation curricula for sentence- and paragraph-level annotation tasks, defines an ordering strategy, and identifies well-performing heuristics and interactively trained models on three existing English datasets. Finally, we provide a proof of concept for annotation curricula in a carefully designed user study with 40 voluntary participants who are asked to identify the most fitting misconception for English tweets about the Covid-19 pandemic. The results indicate that using a simple heuristic to order instances can already significantly reduce the total annotation time while preserving a high annotation quality. Annotation curricula thus can be a promising research direction to improve data collection. To facilitate future research—for instance, to adapt annotation curricula to specific tasks and expert annotation scenarios—all code and data from the user study consisting of 2,400 annotations is made available

show abstract

Section: Masked Language Modeling Loss (Mlm)mentioning

confidence: 99%

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee

Klie

Gurevych

2022

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Instead of switching between different strategies, adaptive estimators may provide another way to consider different objectives when selecting instances for annotation. As shown by Lee, Meyer, and Gurevych (2020) for language learning exercises, it may be possible to sample instances that jointly suffice seemingly counteracting objectives such as reducing the overall annotation time while being preferable for model training. We will investigate such strategies in future work.…”

Section: Limitations and Future Workmentioning

confidence: 99%

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee¹,

Klie²,

Gurevych³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations; especially in citizen science or crowd sourcing scenarios where domain expertise is not required and only annotation guidelines are provided. To alleviate these issues, this work proposes annotation curricula, a novel approach to implicitly train annotators. The goal is to gradually introduce annotators into the task by ordering instances that are annotated according to a learning curriculum. To do so, this work formalizes annotation curricula for sentence-and paragraph-level annotation tasks, defines an ordering strategy, and identifies well-performing heuristics and interactively trained models on three existing English datasets. Finally, a user study is conducted with 40 voluntary participants who are asked to identify the most fitting misconception for English tweets about the Covid-19 pandemic. The results show that using a simple heuristic to order instances can already significantly reduce the total annotation time while preserving a high annotation quality. Annotation curricula thus can provide a novel way to improve data collection. To facilitate future research, all code and data from the user study consisting of 2,400 annotations is made available. 1

show abstract

“…However, the interaction with a not yet fully trained model can get monotonous or can lead to frustration on the part of the users if they do not benefit from the interaction themselves (Lee et al, 2020). Wang et al (2016) present an interactive language learning setting, called SHRDLURN, in which a model learns a language by interacting with a player in a game environment, hence making the interactive learning setting more attractive and fun for users.…”

Section: Introductionmentioning

confidence: 99%

Proceedings of the First Workshop on Interactive Learning for Natural Language Processing

2021

View full text Add to dashboard Cite

Biases and artifacts in training data can cause unwelcome behavior in text classifiers (such as shallow pattern matching), leading to lack of generalizability. One solution to this problem is to include users in the loop and leverage their feedback to improve models. We propose a novel explanatory debugging pipeline called HILDIF, enabling humans to improve deep text classifiers using influence functions as an explanation method. We experiment on the Natural Language Inference (NLI) task, showing that HILDIF can effectively alleviate artifact problems in fine-tuned BERT models and result in increased model generalizability.

show abstract

Empowering Active Learning to Jointly Optimize System and User Demands

Cited by 6 publications

References 24 publications

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation Curricula to Implicitly Train Non-Expert Annotators

Proceedings of the First Workshop on Interactive Learning for Natural Language Processing

Contact Info

Product

Resources

About