Challenges of Testing Machine Learning Applications

Huang, Song; Liu, Er-Hu; Hui, Zhanwei; Tang, Shiqi; Zhang, Suojuan

doi:10.23940/ijpe.18.06.p18.12751282

Cited by 12 publications

(12 citation statements)

References 26 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors in [667] described numerous challenges to test several machine learning models. In the military, quick action has to be taken against a message.…”

Section: ) Adversarial Inputs To ML Modelsmentioning

confidence: 99%

A Survey on Machine Learning Techniques for Cyber Security in the Last Decade

et al. 2020

View full text Add to dashboard Cite

Pervasive growth and usage of the Internet and mobile applications have expanded cyberspace. The cyberspace has become more vulnerable to automated and prolonged cyberattacks. Cyber security techniques provide enhancements in security measures to detect and react against cyberattacks. The previously used security systems are no longer sufficient because cybercriminals are smart enough to evade conventional security systems. Conventional security systems lack efficiency in detecting previously unseen and polymorphic security attacks. Machine learning (ML) techniques are playing a vital role in numerous applications of cyber security. However, despite the ongoing success, there are significant challenges in ensuring the trustworthiness of ML systems. There are incentivized malicious adversaries present in the cyberspace that are willing to game and exploit such ML vulnerabilities. This paper aims to provide a comprehensive overview of the challenges that ML techniques face in protecting cyberspace against attacks, by presenting a literature on ML techniques for cyber security including intrusion detection, spam detection, and malware detection on computer networks and mobile networks in the last decade. It also provides brief descriptions of each ML method, frequently used security datasets, essential ML tools, and evaluation metrics to evaluate a classification model. It finally discusses the challenges of using ML techniques in cyber security. This paper provides the latest extensive bibliography and the current trends of ML in cyber security.

show abstract

“…Authors in [667] described numerous challenges to test several machine learning models. In the military, quick action has to be taken against a message.…”

Section: ) Adversarial Inputs To ML Modelsmentioning

confidence: 99%

A Survey on Machine Learning Techniques for Cyber Security in the Last Decade

et al. 2020

View full text Add to dashboard Cite

show abstract

“…A total of 12 papers studied the challenges in software testing for AI. Nine of them discussed the challenges, issues, and needs in AI software testing based on the current state of the art, either for generic AI systems [29,62,81,150,175] or focusing on the particular challenges for autonomous vehicles or other safety-critical systems [105,117,119,170]. Finally, three proposals identified the challenges for generic AI or ML systems through empirical methods like questionnaire surveys with practitioners [79,88,224].…”

Section: Software Testing (115 Studies)mentioning

confidence: 99%

“…List of references Software Engineering Models and Methods [20,62,72,79,88,92,105,119,140,175,182] Software Requirements [80,88,92,202,224] Software Testing [29,62,81,150,172,175,182,224] Software Quality [62,79,80,88,92,113,135,172,182] Software Engineering Professional Practice [84,88,172,202] Software Construction [80,88,92,116,124,134,136,172,175,224 • Elicitation. The use of data as a source of requirements is attractive, but due to the high volume of such data, it requires tool support to detect features from massive data.…”

Section: Swebok Knowledge Areamentioning

confidence: 99%

Software Engineering for AI-Based Systems: A Survey

Martínez-Fernández,

Bogner,

Franch

et al. 2021

Preprint

View full text Add to dashboard Cite

AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image-and speechrecognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on Software Engineering (SE) approaches for building, operating, and maintaining AI-based systems.To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state of the art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula. CCS Concepts: • Software and its engineering → Software creation and management; • Computing methodologies → Machine learning;

show abstract

“…In this paper, we report about techniques and tools currently used by practitioners to cope with these challenges. Song Huang et al [36] investigate the characteristics of Naive Bayesian classifier and DNN classifier and analyze the testing challenges of machine learning applications like: Generating reliable test oracles, Generating effective corner cases, Improving test coverage and Testing the ML applications with millions of parameters. Then some initial techniques were suggested for machine learning applications which use Naive Bayesian classifier and DNN classifier to mitigate these challenges.…”

Section: Related Workmentioning

confidence: 99%

Machine Learning Application Development: Practitioners' Insights

Rahman¹,

Khomh²,

Hamidi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Nowadays, intelligent systems and services are getting increasingly popular as they provide data-driven solutions to diverse real-world problems, thanks to recent breakthroughs in Artificial Intelligence (AI) and Machine Learning (ML). However, machine learning meets software engineering not only with promising potentials but also with some inherent challenges. Despite some recent research efforts, we still do not have a clear understanding of the challenges of developing ML-based applications and the current industry practices. Moreover, it is unclear where software engineering researchers should focus their efforts to better support ML application developers. In this paper, we report about a survey that aimed to understand the challenges and best practices of ML application development. We synthesize the results obtained from 80 practitioners (with diverse skills, experience, and application domains) into 17 findings; outlining challenges and best practices for ML ap-

show abstract

Challenges of Testing Machine Learning Applications

Cited by 12 publications

References 26 publications

A Survey on Machine Learning Techniques for Cyber Security in the Last Decade

A Survey on Machine Learning Techniques for Cyber Security in the Last Decade

Software Engineering for AI-Based Systems: A Survey

Machine Learning Application Development: Practitioners' Insights

Contact Info

Product

Resources

About