ConfLabeling: Assisting Image Labeling with User and System Confidence

Lu, Yi; Chang, Chia–Ming; Igarashi, Takeo

doi:10.1007/978-3-031-21707-4_26

Cited by 2 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach further reduces the need for multiple persons per image. In addition, we found that the reviewers can record uncertainty as part of the labeling task (as also suggested in Lu et al (2022)). This recorded uncertainty can be used for additional control of the most difficult analysis or assist researchers in analyzing the transcribed data.…”

Section: Evaluation Of the Annotation Workflowsupporting

confidence: 58%

More Efficient Manual Review of Automatically Transcribed Tabular Data

Pedersen,

Johansen,

Holsbø

et al. 2024

hlcs

View full text Add to dashboard Cite

Any machine learning method for transcribing historical text requires manual verification and correction, which is often time-consuming and expensive. Our aim is to make it more efficient. Previously, we developed a machine learning model to transcribe 2.3 million handwritten occupation codes from the Norwegian 1950 census. Here, we manually review the 90,000 codes (3%) for which our model had the lowest confidence scores. We allocated these codes to human reviewers, who used our custom annotation tool to review them. The reviewers agreed with the model's labels 31.9% of the time. They corrected 62.8% of the labels, and 5.1% of the images were uncertain or assigned invalid labels. 9,000 images were reviewed by multiple reviewers, resulting in an agreement of 86.4% and a disagreement of 9%. The results suggest that one reviewer per image is sufficient. We recommend that reviewers indicate any uncertainty about the label they assign to an image by adding a flag to their label. Our interviews show that the reviewers performed internal quality control and found our custom tool to be useful and easy to operate. We provide guidelines for efficient and accurate transcription of historical text by combining machine learning and manual review. We have open-sourced our custom annotation tool and made the reviewed images open access.

show abstract

Section: Evaluation Of the Annotation Workflowsupporting

confidence: 58%