Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

Liao, Yuan-Hong; Kar, Amlan; Fidler, Sanja

doi:10.1109/cvpr46437.2021.00433

Cited by 12 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The semi-automated approach is where a certain level of automation is used along with manual annotations. The majority of publicly available datasets use manual or semi-automated annotations [26]. These datasets are annotated by a group of academics, contributors, and crowd-sourcing labor with a certain level of automation.…”

Section: Data Preparationmentioning

confidence: 99%

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Kamath,

A.,

Kini

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Most computer vision applications that use deep learning on constrained device come from the Internet of Things (IoT) or robotics fields, where low-quality cameras are used to capture input images in real time. Since most pretrained models typically undergo training on high-quality image datasets, the low-quality, noisy, or blurry images captured with these resource-constrained devices could possibly have a negative impact on the models' performance. To determine if model performance is impacted by training models using low-quality data, a secondary image dataset named MOD-2022 was prepared for object detection and tracking tasks using an exploratory data preparation methodology. This dataset was primarily designed to include a wider range of classes with an adequate number of images per class while being free of errors due to inaccurate labelling or annotations. Additionally, a training approach is also proposed to support the model's training when the dataset is considerably large. A VGG16-SSD model was trained with this approach on the prepared dataset and deployed on a Raspberry Pi and it showed that this approach is very useful in developing models for resource-constrained applications. Furthermore, this data preparation approach can be extended to prepare numerous other datasets required for training models designed to be deployed on constrained devices similar to the Raspberry Pi.

show abstract

Section: Data Preparationmentioning

confidence: 99%

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Kamath,

A.,

Kini

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Lastly, another related stream of work focuses on efficiently combining predictions of multiple human annotators to derive high-quality labels, e.g., from crowdsourced predictions (Branson, Van Horn, and Perona 2017;Guan et al 2018;Liao, Kar, and Fidler 2021;Welinder and Perona 2010). Whereas these approaches focus on ground truth label quality using a larger worker pool, we are interested in learning the capabilities of an individual human expert only from a minimal number of expert predictions.…”

Section: Related Workmentioning

confidence: 99%

Learning to Defer with Limited Expert Predictions

Hemmer

Thede

Vössing

et al. 2023

AAAI

View full text Add to dashboard Cite

Recent research suggests that combining AI models with a human expert can exceed the performance of either alone. The combination of their capabilities is often realized by learning to defer algorithms that enable the AI to learn to decide whether to make a prediction for a particular instance or defer it to the human expert. However, to accurately learn which instances should be deferred to the human expert, a large number of expert predictions that accurately reflect the expert's capabilities are required—in addition to the ground truth labels needed to train the AI. This requirement shared by many learning to defer algorithms hinders their adoption in scenarios where the responsible expert regularly changes or where acquiring a sufficient number of expert predictions is costly. In this paper, we propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. It encompasses (1) the training of an embedding model with ground truth labels to generate feature representations that serve as a basis for (2) the training of an expertise predictor model to approximate the expert's capabilities. (3) The expertise predictor generates artificial expert predictions for instances not yet labeled by the expert, which are required by the learning to defer algorithms. We evaluate our approach on two public datasets. One with "synthetically" generated human experts and another from the medical domain containing real-world radiologists' predictions. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions. Furthermore, we demonstrate that even a small number of expert predictions per class is sufficient for these algorithms to exceed the performance the AI and the human expert can achieve individually.

show abstract

“…These label annotation procedures are astonishingly onerous, a problem that we can attenuate by using some proposed tools and methods in the literature (Acuna et al, 2018;Ling et al, 2019;Liao et al, 2021), such as the human-inthe-loop (Wu et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

OdontoAI: A human-in-the-loop labeled data set and an online platform to boost research on dental panoramic radiographs

Silva¹,

Pinheiro²,

Sobrinho³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning has remarkably advanced in the last few years, supported by large labeled data sets. These data sets are precious yet scarce because of the time-consuming labeling procedures, discouraging researchers from producing them. This scarcity is especially true in dentistry, where deep learning applications are still in an embryonic stage. Motivated by this background, we address in this study the construction of a public data set of dental panoramic radiographs. Our objects of interest are the teeth, which are segmented and numbered, as they are the primary targets for dentists when screening a panoramic radiograph. We benefited from the human-in-the-loop (HITL) concept to ex-

show abstract

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

Cited by 12 publications

References 17 publications

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Learning to Defer with Limited Expert Predictions

OdontoAI: A human-in-the-loop labeled data set and an online platform to boost research on dental panoramic radiographs

Contact Info

Product

Resources

About