CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries

Pham, Hūng Viêt; Lutellier, Thibaud; Qi, Weizhen; Tan, Lin

doi:10.1109/icse.2019.00107

Cited by 117 publications

(55 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ADS relies on ML models that are trained using various deep learning and machine learning frameworks. Prior studies [13,28] found that different frameworks may result in a model with slightly different performance. As shown in Table 2, Apollo developers use ML models that are trained using various frameworks (i.e., CaffeNet, Paddle, PyTorch, and TensorRT).…”

Section: Discussion and Implicationmentioning

confidence: 99%

“…This posts great concern and calls for attention that the dependency libraries of an ML system could also be an important factor that impacts its quality. Some recent work [28] starts to effectively detect the potential quality issues in the framework. When it comes to ADS, Joshua et al [18] perform an early step study to investigate the common bug symptoms and causes.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo

Peng

Yang

Chen

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

Section: Discussion and Implicationmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo

Peng

Yang

Chen

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

“…Some recent efforts have been made to debug DL models [31], and to study DL program bugs [60], library bugs [42] and DL software bugs across different frameworks and platforms [16]. The results of this paper provide a new angle to characterize DL model defects, which could be useful for other quality assurance activities besides testing.…”

Section: Related Workmentioning

confidence: 91%

Towards characterizing adversarial defects of deep learning software from the lens of uncertainty

Zhang

Xie

Ма

et al. 2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

View full text Add to dashboard Cite

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety-and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. Although some testing, adversarial attack and defense techniques have been recently proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty. In this paper, we conduct a large-scale study towards bridging this gap. We first investigate the capability of multiple uncertainty metrics in differentiating benign examples (BEs) and AEs, which enables to characterize the uncertainty patterns of input data. Then, we identify and categorize the uncertainty patterns of BEs and AEs, and find that while BEs and AEs generated by existing methods do follow common uncertainty patterns, some other uncertainty patterns are largely missed. Based on this, we propose an automated testing technique to generate multiple types of uncommon AEs and BEs that are largely missed by existing techniques. Our further evaluation reveals that the uncommon data generated by our

show abstract

“…They play a more important role in ML development than in traditional software development. ML Framework testing thus checks whether the frameworks of machine learning have bugs that may lead to problems in the final system [48]. Bug Detection in Learning Program.…”

Section: Testing Componentsmentioning

confidence: 99%

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

503

318

View full text Add to dashboard Cite

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 128 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms-machine learning, software testing, deep neural network, ! • Jie M. Zhang and Mark Harman are with CREST, University College London, United Kingdom. Mark Harman is also with Facebook London.

show abstract

CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries

Cited by 117 publications

References 46 publications

A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo

A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo

Towards characterizing adversarial defects of deep learning software from the lens of uncertainty

Machine Learning Testing: Survey, Landscapes and Horizons

Contact Info

Product

Resources

About