Toward Human-in-the-Loop Collaboration Between Software Engineers and Machine Learning Algorithms

Nascimento, Nathalia; Alencar, Paulo; Lucena, Carlos José Pereira de; Cowan, Donald

doi:10.1109/bigdata.2018.8622107

Cited by 13 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These two heads are ML methods and humans. For example, in Nascimento et al [ 3 ], the combination of software engineers of different levels of expertise as human experts and ML methods was investigated and compared for a case study on streetlight automation planning. The results showed that human experts outperform the ML methods in some experimental conditions, but the ML methods have better performance in other conditions.…”

Section: Introductionmentioning

confidence: 99%

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

Maadi

Khorshidi

Aickelin

2021

IJERPH

View full text Add to dashboard Cite

Objective: To provide a human–Artificial Intelligence (AI) interaction review for Machine Learning (ML) applications to inform how to best combine both human domain expertise and computational power of ML methods. The review focuses on the medical field, as the medical ML application literature highlights a special necessity of medical experts collaborating with ML approaches. Methods: A scoping literature review is performed on Scopus and Google Scholar using the terms “human in the loop”, “human in the loop machine learning”, and “interactive machine learning”. Peer-reviewed papers published from 2015 to 2020 are included in our review. Results: We design four questions to investigate and describe human–AI interaction in ML applications. These questions are “Why should humans be in the loop?”, “Where does human–AI interaction occur in the ML processes?”, “Who are the humans in the loop?”, and “How do humans interact with ML in Human-In-the-Loop ML (HILML)?”. To answer the first question, we describe three main reasons regarding the importance of human involvement in ML applications. To address the second question, human–AI interaction is investigated in three main algorithmic stages: 1. data producing and pre-processing; 2. ML modelling; and 3. ML evaluation and refinement. The importance of the expertise level of the humans in human–AI interaction is described to answer the third question. The number of human interactions in HILML is grouped into three categories to address the fourth question. We conclude the paper by offering a discussion on open opportunities for future research in HILML.

show abstract

Section: Introductionmentioning

confidence: 99%

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

Maadi

Khorshidi

Aickelin

2021

IJERPH

View full text Add to dashboard Cite

show abstract

“…ML allows systems to perform human-centric activities at a much larger scale [48]. In fact, an empirical study [49] has been conducted to understand whether software engineers can utilize ML techniques for the improvement of their SE process and whether solutions proposed by engineers still outperform ML techniques. However, the need for ML techniques is still pertinent due to their ability to outperform in most SE activities.…”

Section: Applications Of ML Aiming At Software Maintenancementioning

confidence: 99%

“…Despite the intriguing tendency of full automation, complete automation could often result in a potentially fallible system, therefore, practitioners are encouraged to employ ML techniques with humans in the loop wherever there is a presence of criticality [1,49]. In addition, there is a significant lack of studies showing the cost-benefit analysis of their proposed ML techniques, which would be vital for ML-based approaches to be feasible for adaptation in the industry.…”

Section: Applications Of ML Aiming At Software Maintenancementioning

confidence: 99%

“…The important thing is that the found studies are a good representation of the population, which we ensured in this study by adopting a rigorous paper selection process. All Stages Approach/Technique/Method RNN [63] All Stages Other Other [64] All Stages Other Other [56] All Stages Comparative Analysis Other [57] All Stages Other Other [69] All Stages Other RNN, RBM [14] All Stages Model/Framework Other [67] All Stages Tool RF [49] All Stages Other Other [62] All Stages Tool NLP [61] All Stages Other DT [48] All Stages Other Other [65] All Stages Other Other [1] All Stages Other Other [60] All Stages Other Other [68] All Stages Comparative Analysis LR, SVM, NB [66] All Stages Approach/Technique/Method LSTM [11] All Stages Other Other [59] All Stages Other Other [21] Requirements Approach/Technique/Method NB, KNN, RF [70] Requirements Approach/Technique/Method SVM, SMO, NB [82] Requirements Approach/Technique/Method PN [75] Requirements Model/Framework ProbPoly [71] Requirements Approach/Technique/Method Text2Model [84] Requirements Approach/Technique/Method RF [72] Requirements Approach/Technique/Method Other [22] Requirements Approach/Technique/Method NB, RF, LR, SGD, DT [23] Requirements Approach/Technique/Method Boosting [81] Requirements Approach/Technique/Method NSGA-II algorithm [24] Requirements Model/Framework CNN [73] Requirements Approach/Technique/Method FL [76] Requirements Approach/Technique/Method LP, SMO, NB, KNN [86] Requirements Approach/Technique/Method J48, FSS, CFS [25] Requirements Model/Framework RNN [85] Requirements Model/Framework KNN [...…”

Section: A External Validitymentioning

confidence: 99%

See 1 more Smart Citation

A Literature Review of Using Machine Learning in Software Development Life Cycle Stages

Shafiq¹,

Mashkoor

Mayr-Dorn

et al. 2021

IEEE Access

View full text Add to dashboard Cite

The software engineering community is rapidly adopting machine learning for transitioning modern-day software towards highly intelligent and self-learning systems. However, the software engineering community is still discovering new ways how machine learning can offer help for various software development life cycle stages. In this article, we present a study on the use of machine learning across various software development life cycle stages. The overall aim of this article is to investigate the relationship between software development life cycle stages, and machine learning tools, techniques, and types. We attempt a holistic investigation in part to answer the question of whether machine learning favors certain stages and/or certain techniques.

show abstract

“…Of the 53 papers, we selected the 23 most relevant. We observed that the greater part of the papers was filtered out because they describe solutions of ML to SE, as in the article of Nascimento et al [19].…”

Section: Search Strategymentioning

confidence: 99%

Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review

Lorenzoni¹,

Alencar²,

Nascimento³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Data scientists often develop machine learning models to solve a variety of problems in the industry and academy but not without facing several challenges in terms of Model Development. The problems regarding Machine Learning Development involves the fact that such professionals do not realize that they usually perform ad-hoc practices that could be improved by the adoption of activities presented in the Software Engineering Development Lifecycle. Of course, since machine learning systems are different from traditional Software systems, some differences in their respective development processes are to be expected. In this context, this paper is an effort to investigate the challenges and practices that emerge during the development of ML models from the software engineering perspective by focusing on understanding how software developers could benefit from applying or adapting the traditional software engineering process to the Machine Learning workflow.

show abstract

Toward Human-in-the-Loop Collaboration Between Software Engineers and Machine Learning Algorithms

Cited by 13 publications

References 9 publications

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

A Literature Review of Using Machine Learning in Software Development Life Cycle Stages

Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review

Contact Info

Product

Resources

About