Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface

Desmond, Michael; Müller, Michael; Ashktorab, Zahra; Dugan, Casey; Duesterwald, Evelyn; Brimijoin, Kristina; Finegan-Dollak, Catherine; Brachman, Michelle; Sharma, Aabhas; Joshi, Narendra Nath; Qian, Pan

doi:10.1145/3397481.3450698

Cited by 29 publications

(19 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In some studies, the quality of a decision (e.g. [20,52]) or the speed of a process (e.g. [19,92]) were improved with the presence of AI support.…”

Section: Human-centered Examinations Of Ai-supported Tasksmentioning

confidence: 99%

“…Lai and Tan [52] found that showing people an AI model's predictions improved their performance in a deception detection task. Desmond et al [20] and Ashktorab et al [6] both found that AI-assistance in data labeling tasks, in which a human annotator made decisions for which labels to apply to data, sped up the data labeling process and increased the accuracy of data labelers.…”

Section: Ai-supported Decisionmentioning

confidence: 99%

“…Physical demand was excluded as it was not applicable to our code translation task. Dimensions were rated on a 20-point scale(1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20), with 20 being the highest-level rating. Linear models were used to compare each TLX dimension and they included terms that controlled for AI presence, task order, data structure, and participant ID as a random effect.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Better Together? An Evaluation of AI-Supported Code Translation

Weisz¹,

Müller²,

Ross³

et al. 2022

27th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.CCS Concepts: • Human-centered computing → HCI theory, concepts and models; • Software and its engineering → Designing software; • Computing methodologies → Generative and developmental approaches.

show abstract

“…In some studies, the quality of a decision (e.g. [20,52]) or the speed of a process (e.g. [19,92]) were improved with the presence of AI support.…”

Section: Human-centered Examinations Of Ai-supported Tasksmentioning

confidence: 99%

Section: Ai-supported Decisionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Better Together? An Evaluation of AI-Supported Code Translation

Weisz¹,

Müller²,

Ross³

et al. 2022

27th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to the use of supervised learning and end-to-end extraction methods [21], the lack of labeled data has introduced the problem of insufficient data accuracy. Moreover, these methods can only process text [5,16,35], use NLP to analyze the tables with the content [21], and do not store the tables as structured data for broader big data analysis.…”

Section: Data Extraction In Scientific Literaturementioning

confidence: 99%

DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance

Shao¹,

Jia²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

extract and aggregate data, we proposed DeepShovel, a publicly-available AI-assisted data extraction system to support their needs.DeepShovel leverages the state-of-the-art neural network models to support researcher(s) easily and accurately annotate papers (in the PDF format) and extract data from tables, figures, maps, etc. in a human-AI collaboration manner. A follow-up user evaluation with 14 researchers suggested DeepShovel improved users' efficiency of data extraction for building scientific databases, and encouraged teams to form a larger scale but more tightly-coupled collaboration.CCS Concepts: • Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Artificial intelligence.

show abstract

“…It requires human labelers to examine data and assign labels in order to create training data for a machine learning task. (Desmond et al 2021) studied the effects of AI assistance on human labeling performance and found that labelers were significantly faster and more accurate when presented with label suggestions generated by a learner trained on previously labeled data. Based on observations from this study we designed a data labeling system which seamlessly integrates a learner into the labeling process.…”

Section: Introductionmentioning

confidence: 99%

AI Assisted Data Labeling with Interactive Auto Label

Desmond¹,

Brachman²,

Duesterwald³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

We demonstrate an AI assisted data labeling system which applies unsupervised and semi-supervised machine learning to facilitate accurate and efficient labeling of large data sets. Our system (1) applies representative data sampling and active learning in order to seed and maintain a semi-supervised learner that assists the human labeler (2) provides visual labeling assistance and optimizes labeling mechanics using predicted labels (3) seamlessly updates and learns from ongoing human labeling activity (4) captures and presents metrics that indicate the quality of labeling assistance, and (5) provides an interactive auto labeling interface to group, review and apply predicted labels in a scalable manner.

show abstract

Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface

Cited by 29 publications

References 37 publications

Better Together? An Evaluation of AI-Supported Code Translation

Better Together? An Evaluation of AI-Supported Code Translation

DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance

AI Assisted Data Labeling with Interactive Auto Label

Contact Info

Product

Resources

About