Oracle or Teacher? A Systematic Overview of Research on Interactive Labeling for Machine Learning

Knaeble, Merlin; Nadj, Mario; Maedche, Alexander

doi:10.30844/wi_2020_a1-knaeble

Cited by 4 publications

(3 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interactive labeling provides model developers with visual interfaces to identify, select, and label interesting instances to improve the labeling flexibility [10]. Many visualization tools integrate new workflows to reduce the samples to be labeled [23][24][25][26].…”

Section: Active Sample Recommendationmentioning

confidence: 99%

See 1 more Smart Citation

Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data

Tan

Huang

2022

Applied Sciences

View full text Add to dashboard Cite

The practice of applying a classifier (called a pattern classifier and abbreviated as PC below) in a visual analysis system to identify patterns from interactively generated visualizations is gradually emerging. Demonstrated cases in existing works focus on ideal scenarios where the analyst can determine all the pattern types in advance without adjusting the classifier settings during the exploration process. However, in most real-world scenarios, analysts know nothing about data patterns before exploring the dataset and inevitably find novel patterns during the exploration. This difference makes the traditional classifier training and application mode less suitable. Analysts have to artificially determine whether each generated visualization contains new data patterns to adjust the classifier setting, thus affecting the automation of the data exploration. This paper proposes a novel PC-based data exploration approach. The core of the approach is an active-learning indicator for automatically identifying visualizations involving new pattern classes. Analysts thus can apply PCs to explore data while dynamically adjusting the PCs using these visualizations. We further propose a PC-based visualization framework that takes full advantage of the PC in terms of efficiency by allowing analysts to explore an exploring space, rather than a single visualization at a time. The results of the quantitative experiment and the performance of participants in the user study demonstrate the effectiveness and usability of the method.

show abstract

Section: Active Sample Recommendationmentioning

confidence: 99%

“…As the analyst generates and reads visualizations, the number of recognized patterns gradually increases. Even though we can initialize a PC through manual sampling and labeling [10][11][12][13], the PC inevitably classifies a newly generated visualization that contains never-encountered patterns into an existing class during IDE.…”

Section: Introductionmentioning

confidence: 99%

Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data

Tan

Huang

2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Thereby, researchers can provide basic guidance for workers but workers' interpretation of the data is still required to identify the most suitable code. • R3 Contextualize Coding: Providing context and showing multiple items at once not only improves the quality of coding results [2], but also increases the speed of coding [34]. The system shall provide workers context for coding the data.…”

Section: System Requirementsmentioning

confidence: 99%

Accelerating Deductive Coding of Qualitative Data: An Experimental Study on the Applicability of Crowdsourcing

Haug

Rietz

Maedche

2021

Mensch Und Computer 2021

Self Cite

View full text Add to dashboard Cite

Figure 1: Our interactive coding system enables a crowd of non-experts to code semi-structured qualitative data. (1) First, the workers code the primary topics of all interview answers. (2) Then, the workers code each interview answer with the respective specific codes in separate tasks for each primary topic. (3) Finally, the workers' most mentioned code is set as final label. The agreement among workers indicates the crowds' consistency and the agreement between the workers' and experts' final label shows the accuracy of the crowd.

show abstract

AnnoVate: Revolutionizing Data Annotation with Automated Labeling Technique

Qazi,

Muhammad Naseem,

Aslam

et al. 2024

VFAST trans. softw. eng.

View full text Add to dashboard Cite

This research introduces AnnoVate, an innovative web application designed to automate the labor-intensive task of object annotation for computer vision applications. Focused on image annotation, the study addresses the escalating demand for data refinement and labeling in the field of artificial intelligence (AI). Leveraging the power of YOLOv8 (You Only Look Once), a high-performance object detection algorithm, AnnoVate minimizes human intervention while achieving an impressive 85% overall accuracy in object detection. The methodology integrates active learning, allowing labelers to selectively prioritize uncertain data during the labeling process. An iterative training approach continuously refines the model, creating a self-improving loop that enhances accuracy over successive loops. The system's flexibility enables users to export labeled datasets for their preferred AI model architectures. AnnoVate not only overcomes the limitations of traditional labeling methods but also establishes a collaborative human-machine interaction paradigm, setting the stage for further advancements in computer vision.

show abstract

Oracle or Teacher? A Systematic Overview of Research on Interactive Labeling for Machine Learning

Cited by 4 publications

References 51 publications

Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data

Active Pattern Classification for Automatic Visual Exploration of Multi-Dimensional Data

Accelerating Deductive Coding of Qualitative Data: An Experimental Study on the Applicability of Crowdsourcing

AnnoVate: Revolutionizing Data Annotation with Automated Labeling Technique

Contact Info

Product

Resources

About