DiffChaser: Detecting Disagreements for Deep Neural Networks

Xie, Xiaofei; Ма, Лей; Wang, Haijun; Li, Yuekang; Liu, Yang; Li, Xiaohong

doi:10.24963/ijcai.2019/800

Cited by 71 publications

(56 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, generating such specific inputs is not the primary concern of WebExplor, and all advanced input generation techniques can be incorporated in the WebExplor to further enhance the performance. Recently, there are also some research on testing deep learning models [49,58,59,60], which could be used to test reinforcement learning models. Reinforcement Learning Based Testing.…”

Section: F Threats To Validitymentioning

confidence: 99%

Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Yan

Liu

Xie

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

Self Cite

View full text Add to dashboard Cite

Web testing has long been recognized as a notoriously difficult task. Even nowadays, web testing still heavily relies on manual efforts while automated web testing is far from achieving human-level performance. Key challenges in web testing include dynamic content update and deep bugs hiding under complicated user interactions and specific input values, which can only be triggered by certain action sequences in the huge search space. In this paper, we propose WebExplor, an automatic end-to-end web testing framework, to achieve an adaptive exploration of web applications. WebExplor adopts curiosity-driven reinforcement learning to generate high-quality action sequences (test cases) satisfying temporal logical relations. Besides, WebExplor incrementally builds an automaton during the online testing process, which provides high-level guidance to further improve the testing efficiency. We have conducted comprehensive evaluations of WebExplor on six real-world projects, a commercial SaaS web application, and performed an in-thewild study of the top 50 web applications in the world. The results demonstrate that in most cases WebExplor can achieve significantly higher failure detection rate, code coverage and efficiency than existing state-of-the-art web testing techniques. WebExplor also detected 12 previously unknown failures in the commercial web application, which have been confirmed and fixed by the developers. Furthermore, our in-the-wild study further uncovered 3,466 exceptions and errors.

show abstract

Section: F Threats To Validitymentioning

confidence: 99%

Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Yan

Liu

Xie

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Classic testing methodologies have also been incorporated for DNN testing, including differential testing [41], coverage-guided testing [38,52,55], mutation testing [30] and concolic testing [48]. Some advanced test generation methods [7,56,58] have also been proposed to achieve better testing for different applications. Similar to samples generated by adversarial attacks, it still lacks a study on the relationship of samples generated by testing techniques and uncertainty.…”

Section: Related Workmentioning

confidence: 99%

Towards characterizing adversarial defects of deep learning software from the lens of uncertainty

Zhang

Xie

Ма

et al. 2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Self Cite

View full text Add to dashboard Cite

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety-and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. Although some testing, adversarial attack and defense techniques have been recently proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty. In this paper, we conduct a large-scale study towards bridging this gap. We first investigate the capability of multiple uncertainty metrics in differentiating benign examples (BEs) and AEs, which enables to characterize the uncertainty patterns of input data. Then, we identify and categorize the uncertainty patterns of BEs and AEs, and find that while BEs and AEs generated by existing methods do follow common uncertainty patterns, some other uncertainty patterns are largely missed. Based on this, we propose an automated testing technique to generate multiple types of uncommon AEs and BEs that are largely missed by existing techniques. Our further evaluation reveals that the uncommon data generated by our

show abstract

“…Unlike ordinary programs, neural networks require new testing criteria [145]. However, using greybox fuzzing to detect bugs in neural networks seems to be viable and it is a promising direction [146][147][148]. These existing works are just introducing greybox fuzzing to the area of neural networks.…”

Section: Resultsmentioning

confidence: 99%

Semantic-based vulnerability analysis of software

Li¹

Self Cite

View full text Add to dashboard Cite

The contributions of the co-authors are as follows: • Hongxu and I were the lead authors. We share equal contribution. We wrote the code and conducted the experiments together. I received a lot of help and love during my Ph.D. study from many people. No words can fully express my thankfulness but I do want to thank some of them explicitly here. First, I would like to thank my supervisor Prof. Yang Liu for his precious guidance and constructive feedback. His passion for research and work has influenced me a lot. His knowledge and experience provides me guidance and courage to solve different research problems. His selfless support helped me to get through the difficult periods. Second, I would like to specially thank my senior Hongxu Chen. He is very knowledgeable and generous. I have collaborated with him for many works and he has shown admirable perseverance and talents during research. I would also like to thank Dr. Alwen Tiu, Dr. Bihuan Chen and Dr. Lei Wei as they are the very first people who helped me during my Ph.D. study. Their valuable support and help have guided me to get on the right track of research.

show abstract

DiffChaser: Detecting Disagreements for Deep Neural Networks

Cited by 71 publications

References 11 publications

Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Automatic Web Testing Using Curiosity-Driven Reinforcement Learning

Towards characterizing adversarial defects of deep learning software from the lens of uncertainty

Semantic-based vulnerability analysis of software

Contact Info

Product

Resources

About