How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey

Ishikawa, Fuyuki; Yoshioka, Nobukazu

doi:10.1109/cesser-ip.2019.00009

Cited by 69 publications

(38 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-Survey papers on the data or model management for ML applications [23,26,27]. -Papers discussing the SE challenges for ML applications [17,18,21,59,83] or the challenges in an application domain [90]. 4.5.…”

Section: Challengesmentioning

confidence: 99%

“…-Risk Management: Risk management of the development, deployment and operation of ML applications is critical, but is rendered difficult by various uncertainties [17,24]. -Effort Estimation: Estimating the effort of an ML project is challenging because it is difficult to know to what extent the ML model will achieve its goal, and to estimate how many iterations will be needed to reach the state in which the performance gets acceptable levels [18,83]. -Corporate Compliance: In a real-world ML application project for a company, the development, deployment and operation may be severely affected by the effort of complying with the privacy policy of the organization and the legal framework.…”

Section: Challengesmentioning

confidence: 99%

“…Patel et al [38] argues that "it is clear that non-expert tools need to support the entire exploratory and iterative process of applying statistical machine learning algorithms." Ishikawa et al [83] reported the difficulties to make customers better understand the properties of ML applications such as imperfections. Trial-based processes can address these difficulties, but further researches are needed to build a solid foundation for the engineering disciplines.…”

Section: Challengesmentioning

confidence: 99%

See 2 more Smart Citations

Software engineering challenges for machine learning applications: A literature review

Kumeno¹

2020

IDT

View full text Add to dashboard Cite

Machine learning techniques, especially deep learning, have achieved remarkable breakthroughs over the past decade. At present, machine learning applications are deployed in many fields. However, the outcomes of software engineering researches are not always easily utilized in the development and deployment of machine learning applications. The main reason for this difficulty is the many differences between machine learning applications and traditional information systems. Machine learning techniques are evolving rapidly, but face inherent technical and non-technical challenges that complicate their lifecycle activities. This review paper attempts to clarify the software engineering challenges for machine learning applications that either exist or potentially exist by conducting a systematic literature collection and by mapping the identified challenge topics to knowledge areas defined by the Software Engineering Body of Knowledge (Swebok).

show abstract

Section: Challengesmentioning

confidence: 99%

Section: Challengesmentioning

confidence: 99%

Section: Challengesmentioning

confidence: 99%

See 1 more Smart Citation

Software engineering challenges for machine learning applications: A literature review

Kumeno¹

2020

IDT

View full text Add to dashboard Cite

show abstract

“…Instead, most ML approaches generate rules based on a set of examples (training data) and a specific fitness function. In addition, a recent survey suggests that Requirements Engineering (RE) is the most difficult activity for the development of ML-based systems [2].…”

Section: Introductionmentioning

confidence: 99%

Requirements Engineering for Machine Learning: Perspectives from Data Scientists

Vogelsang

Borg

2019

2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)

155

View full text Add to dashboard Cite

Machine learning (ML) is used increasingly in realworld applications. In this paper, we describe our ongoing endeavor to define characteristics and challenges unique to Requirements Engineering (RE) for ML-based systems. As a first step, we interviewed four data scientists to understand how ML experts approach elicitation, specification, and assurance of requirements and expectations. The results show that changes in the development paradigm, i.e., from coding to training, also demands changes in RE. We conclude that development of ML systems demands requirements engineers to: (1) understand ML performance measures to state good functional requirements, (2) be aware of new quality requirements such as explainability, freedom from discrimination, or specific legal requirements, and (3) integrate ML specifics in the RE process. Our study provides a first contribution towards an RE methodology for ML systems.

show abstract

“…Such functions implemented by ML components often directly constitute the core functions of the whole system, quality of which is thus affected by the nature of ML. In one survey, more than 40% of the survey participants answered existing approaches do not work for quality assurance of ML-based AI systems [2].…”

Section: Introductionmentioning

confidence: 99%

Adapting SQuaRE for Quality Assessment of Artificial Intelligence Systems

Kuwajima

Ishikawa

2019

2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)

Self Cite

View full text Add to dashboard Cite

More and more software practitioners are tackling towards industrial applications of artificial intelligence (AI) systems, especially those based on machine learning (ML). However, many of existing principles and approaches to traditional systems do not work effectively for the system behavior obtained by training not by logical design. In addition, unique kinds of requirements are emerging such as fairness and explainability. To provide clear guidance to understand and tackle these difficulties, we present an analysis on what quality concepts we should evaluate for AI systems. We base our discussion on ISO/IEC 25000 series, known as SQuaRE, and identify how it should be adapted for the unique nature of ML and Ethics guidelines for trustworthy AI from European Commission. We thus provide holistic insights for quality of AI systems by incorporating the ML nature and AI ethics to the traditional software quality concepts.

show abstract

How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey

Cited by 69 publications

References 10 publications

Software engineering challenges for machine learning applications: A literature review

Software engineering challenges for machine learning applications: A literature review

Requirements Engineering for Machine Learning: Perspectives from Data Scientists

Adapting SQuaRE for Quality Assessment of Artificial Intelligence Systems

Contact Info

Product

Resources

About