A Survey of Human-in-the-loop for Machine Learning

Abstract

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish some tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field, along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.

Integrating Human-in-the-loop into Swarm Learning for Decentralized Fake News Detection

“…The proposed decentralize method is to combine the advantages of swarm learning [23] and human-in the loop (HITL) [24], which is able to accomplish decentralized fake news detection via human feedback and model update in the swarm learning.…”

“…Human-in-the-loop can be applied to improve the performance of machine learning models by integrating human knowledge and experience for data analytics [24]. For example, human can significantly reduce algorithm bias in the training and inference in terms of human feedback for various tasks in the field of natural language processing (NLP) such as text classification [27], syntactic and semantic parsing [28], topic modeling [29], text summarization [30], and sentiment analysis [31].…”

“…Thus, it allows users at the nodes to maintain the confidentiality and privacy of the raw data. Nevertheless, swarm learning is not designed to leverage valuable user feedback on the inference to further enhance learning performance since the user feedback has been approved to effective to improve data analysis in the loop of learning and inference [24].…”

“…In this paper, we propose a human-in-the-loop based based swarm learning (HBSL) to integrate user feedback into the learning and inference of swarm learning via human-in-theloop (HITL) techniques [24]. HITL is to involve human activities in the process of building machine learning models to improve the model performance via human knowledge [24]. We applied HITL to generate the user feedback in the learning process to improve fake news detection.…”

Multi-channel Attentive Graph Convolutional Network With Sentiment Fusion For Multimodal Sentiment Analysis

“…Sentiment analysis (SA) is one of the widely studied fields in the Natural Language Process (NLP), which aims to predict the individual's attitude towards a product or event [1]. With the rapid development of Internet technology, the Multimodal Sentiment Analysis (MSA) increasingly attracts more attention in recent years [2]. The goal of MSA is to determine the sentiment of a video, an image or a text-based on multiple modal features (see Fig.…”