Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg, Rickard; Dhingra, Bhuwan; Jun, Yang

doi:10.1145/3544548.3581431

Cited by 3 publications

(2 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Past studies have developed intuitive interfaces that simplify the task of labeling, incorporating features like drag-and-drop [55] or batch labeling [5], highlighting [19], and leveraging languagebased models [7]. Stureborg et al grouped more similar contents together and different kinds of pass logic to coordinate between the crowdsourced annotators in multi-labeling tasks [56]. Gooding et al performed a comparative study on annotation interfaces for summarization tasks trained in-house annotators with backgrounds of different proficiency levels [25].…”

Section: Related Workmentioning

confidence: 99%

“…In that sense, the two notions can be applied to the interface to help annotators apply the "divide-and-conquer" approach in annotation. This line of designs might consider implementing a step-wise [56], where the workflows will guide annotators through a sequence of stages. For instance, In the first step, annotators can focus on identifying the hidden context and highlighting ambiguous elements.…”

Section: Design Directionsmentioning

confidence: 99%

See 1 more Smart Citation

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Ara,

Salemi,

Hong

et al. 2024

Proceedings of the 29th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoningbased interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators. CCS CONCEPTS• Human-centered computing → Empirical studies in interaction design; User interface design.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Design Directionsmentioning

confidence: 99%

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Ara,

Salemi,

Hong

et al. 2024

Proceedings of the 29th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

Hierarchical Multi-label Classification of Online Vaccine Concerns

Qinyu Zhu,

Stureborg,

Dhingra

2024

Studies in Computational Intelligence

View full text Add to dashboard Cite

Uncovering labeler bias in machine learning annotation tasks

Haliburton,

Leusmann,

Welsch

et al. 2024

AI Ethics

View full text Add to dashboard Cite

As artificial intelligence becomes increasingly pervasive, it is essential that we understand the implications of bias in machine learning. Many developers rely on crowd workers to generate and annotate datasets for machine learning applications. However, this step risks embedding training data with labeler bias, leading to biased decision-making in systems trained on these datasets. To characterize labeler bias, we created a face dataset and conducted two studies where labelers of different ethnicity and sex completed annotation tasks. In the first study, labelers annotated subjective characteristics of faces. In the second, they annotated images using bounding boxes. Our results demonstrate that labeler demographics significantly impact both subjective and accuracy-based annotations, indicating that collecting a diverse set of labelers may not be enough to solve the problem. We discuss the consequences of these findings for current machine learning practices to create fair and unbiased systems.

show abstract

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Cited by 3 publications

References 53 publications

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Hierarchical Multi-label Classification of Online Vaccine Concerns

Uncovering labeler bias in machine learning annotation tasks

Contact Info

Product

Resources

About