AI-Assisted Human Labeling

Ashktorab, Zahra; Desmond, Michael; Andrés, Josh; Müller, Michael; Joshi, Narendra Nath; Brachman, Michelle; Sharma, Aabhas; Brimijoin, Kristina; Qian, Pan; Wolf, Christine T.; Duesterwald, Evelyn; Dugan, Casey; Geyer, Werner; Reimer, Darrell

doi:10.1145/3449163

Cited by 23 publications

(7 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, several domain experts described scenarios where a review may entail working with thousands of documents and/or tight timelines, making it impossible to manually verify all document details. The strategies we observed with Marco for cross-checking and verification could also become ineffective in these scenarios and potentially encourage overreliance [5]. Future work can investigate additional interactions that provide guardrails and mitigate risk in these high-stakes scenarios, for example by providing explorable uncertainty visualizations or further scaffolding results through clustering [5,18,26,46].…”

Section: Discussionmentioning

confidence: 98%

Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Fok,

Lipka,

Sun

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Knowledge workers often need to extract and analyze information from a collection of documents to solve complex information tasks in the workplace, e.g., hiring managers reviewing resumes or analysts assessing risk in contracts. However, foraging for relevant information can become tedious and repetitive over many documents and criteria of interest. We introduce Marco, a mixedinitiative workspace supporting sensemaking over diverse business document collections. Through collection-centric assistance, Marco reduces the cognitive costs of extracting and structuring information, allowing users to prioritize comparative synthesis and decision making processes. Users interactively communicate their information needs to an AI assistant using natural language and compose schemas that provide an overview of a document collection. Findings from a usability study (n=16) demonstrate that when using Marco, users complete sensemaking tasks 16% more quickly, with less effort, and without diminishing accuracy. A design probe with seven domain experts identifies how Marco can benefit various real-world workflows. CCS CONCEPTS• Human-centered computing → Interactive systems and tools.

show abstract

Section: Discussionmentioning

confidence: 98%

Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Fok,

Lipka,

Sun

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Past studies have developed intuitive interfaces that simplify the task of labeling, incorporating features like drag-and-drop [55] or batch labeling [5], highlighting [19], and leveraging languagebased models [7]. Stureborg et al grouped more similar contents together and different kinds of pass logic to coordinate between the crowdsourced annotators in multi-labeling tasks [56].…”

Section: Related Workmentioning

confidence: 99%

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Ara,

Salemi,

Hong

et al. 2024

Proceedings of the 29th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoningbased interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators. CCS CONCEPTS• Human-centered computing → Empirical studies in interaction design; User interface design.

show abstract

“…uncritically to avoid over-reliance (e.g. as observed in Moroz et al's study of Copilot [51], and discussed more generally in Ashktorab et al [9]) as well as automation bias [45,46,65]. We present the full text of the prompt used for the assistant in Appendix D.…”

Section: Supporting Conversational Interactionmentioning

confidence: 99%

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Ross,

Martinez,

Houde

et al. 2023

Preprint

View full text Add to dashboard Cite

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model's responses. We developed a prototype system -the Programmer's Assistant -in order to explore the utility of conversational interactions grounded in code, as well as software engineers' receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant's capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.CCS Concepts: • Human-centered computing → HCI theory, concepts and models; • Software and its engineering → Designing software; • Computing methodologies → Generative and developmental approaches.

show abstract

AI-Assisted Human Labeling

Cited by 23 publications

References 37 publications

Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Contact Info

Product

Resources

About