Human-in-the-Loop Low-Shot Learning

Wan, Sen; Yimin, Hou; Bao, Feng; Ren, Zhiquan; Dong, Yunfeng; Dai, Qionghai; Deng, Yue

doi:10.1109/tnnls.2020.3011559

Cited by 10 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, researchers mostly use confidence-based methods to obtain key samples. This method plays an irreplaceable role in classification tasks [219,220,221,222]. However, for other tasks (for example, semantic segmentation, regression, and target detection tasks), confidence is not the effect is not so obvious.…”

Section: Discussion and Future Directionsmentioning

confidence: 99%

A Survey of Human-in-the-loop for Machine Learning

Wu,

Xiao,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish some tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field, along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.

show abstract

Section: Discussion and Future Directionsmentioning

confidence: 99%

A Survey of Human-in-the-loop for Machine Learning

Wu,

Xiao,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Adding data [16,45,51,121] Relabeling data [76] Reweighting data [12,64,137] Collecting expert labels [98] Passive observation [69,84,118]…”

Section: Active Data Collectionmentioning

confidence: 99%

Perspectives on Incorporating Expert Feedback into Model Updates

Chen¹,

Bhatt²,

Heidari³

et al. 2022

Preprint

View full text Add to dashboard Cite

Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration of how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observationor domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy, and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey.

show abstract

“…Yue et al [13] proposed a framework named Interventional Few-Shot Learning (IFSL) to address an overlooked deficiency in recent FSL methods. Wan et al [12] proposed a HILL learning algorithm to dynamically reject uncertain predictions and mark them to increase a novel set of categories.…”

Section: Human-in-the-loopmentioning

confidence: 99%

“…DLA is a pixellevel classification task. We follow the work of Wan et al [12] and replace the classification confidence with the new confidence. We choose the Three different confidence (Eq.…”

Section: Validation Of Key Samples Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Human-In-The-Loop Document Layout Analysis

Wu,

Ma,

et al. 2021

Preprint

View full text Add to dashboard Cite

Document layout analysis (DLA) aims to divide a document image into different types of regions. DLA plays an important role in the document content understanding and information extraction systems. Exploring a method that can use less data for effective training contributes to the development of DLA. We consider a Human-in-the-loop (HITL) collaborative intelligence in the DLA. Our approach was inspired by the fact that the HITL push the model to learn from the unknown problems by adding a small amount of data based on knowledge. The HITL select key samples by using confidence. However, using confidence to find key samples is not suitable for DLA tasks. We propose the Key Samples Selection (KSS) method to find key samples in high-level tasks (semantic segmentation) more accurately through agent collaboration, effectively reducing costs. Once selected, these key samples are passed to human beings for active labeling, then the model will be updated with the labeled samples. Hence, we revisited the learning system from reinforcement learning and designed a sample-based agent update strategy, which effectively improves the agent's ability to accept new samples. It achieves significant improvement results in two benchmarks (DSSE-200 (from 77.1% to 86.3%) and CS-150 (from 88.0% to 95.6%)) by using 10% of labeled data.

show abstract

Human-in-the-Loop Low-Shot Learning

Cited by 10 publications

References 20 publications

A Survey of Human-in-the-loop for Machine Learning

A Survey of Human-in-the-loop for Machine Learning

Perspectives on Incorporating Expert Feedback into Model Updates

Human-In-The-Loop Document Layout Analysis

Contact Info

Product

Resources

About