DeepDiagnosis

Wardat, Mohammad; Cruz, Breno Dantas; Le, Wei; Rajan, Hridesh

doi:10.1145/3510003.3510071

Cited by 22 publications

(29 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a software maintenance task, this observation emphasizes the need for automatic testing tools for ML components and ML-based software systems. A number of studies on the automatic bug detection [94,95], bug localization [34], debugging [96], and bug repair [32] in ML-based systems that have been carried out during the last few years showed that there is an increasing need for automatic testing tools for ML-based systems.…”

Section: Discussionmentioning

confidence: 99%

Bugs in machine learning-based systems: a faultload benchmark

et al. 2023

View full text Add to dashboard Cite

Rapid growth of applying Machine Learning (ML) in different domains, especially in safety-critical areas, increases the need for reliable ML components, i.e., a software component operating based on ML. Since corrective maintenance, i.e. identifying and resolving systems bugs, is a key task in the software development process to deliver reliable software components, it is necessary to investigate the usage of ML components, from the software maintenance perspective. Understanding the bugs characteristics and maintenance challenges in ML-based systems can help developers of these systems to identify where to focus maintenance and testing efforts, by giving insights into the most error-prone components, most common bugs, etc. In this paper, we investigate the characteristics of bugs in ML-based software systems and the difference between ML and non-ML bugs from the maintenance viewpoint. We extracted 447,948 GitHub repositories that used one of the three most popular ML frameworks, i.e., TensorFlow, Keras, and PyTorch. After multiple filtering steps, we select the top 300 repositories with the highest number of closed issues. We manually investigate the extracted repositories to exclude non-MLbased systems. Our investigation involved a manual inspection of 386 sampled reported issues in the identified ML-based systems to indicate whether they affect ML components or not. Our analysis shows that nearly half of the real

show abstract

Section: Discussionmentioning

confidence: 99%

Bugs in machine learning-based systems: a faultload benchmark

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Zhang et al [63] introduced a DNN training monitoring and automatic repairing tool called AUTOTRAINER that can fix common training problems, like a slow convergence or fluctuating accuracies. Similarly, a tool called DeepDiagnosis that further improved the repair performance for such bugs, was introduced by Wardat et al [56] It applies dynamic analysis to monitoring and detect errors according to various symptoms. It can fix eight different training problems and can do this more efficient and with a better mance than other tools.…”

Section: Related Workmentioning

confidence: 99%

Semantic-Based Neural Network Repair

Schumi

Sun

2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Recently, neural networks have spread into numerous fields including many safety-critical systems. Neural networks are built (and trained) by programming in frameworks such as TensorFlow and PyTorch. Developers apply a rich set of pre-defined layers to manually program neural networks or to automatically generate them (e.g., through AutoML). Composing neural networks with different layers is error-prone due to the non-trivial constraints that must be satisfied in order to use those layers. In this work, we propose an approach to automatically repair erroneous neural networks. The challenge is in identifying a minimal modification to the network so that it becomes valid. Modifying a layer might have cascading effects on subsequent layers and thus our approach must search recursively to identify a "globally" minimal modification. Our approach is based on an executable semantics of deep learning layers and focuses on four kinds of errors which are common in practice. We evaluate our approach for two usage scenarios, i.e., repairing automatically generated neural networks and manually written ones suffering from common model bugs. The results show that we are able to repair 100% of a set of randomly generated neural networks (which are produced with an existing AI framework testing approach) effectively and efficiently (with an average repair time of 21.08s) and 93.75% of a collection of real neural network bugs (with an average time of 3min 40s). CCS CONCEPTS• Software and its engineering → Formal software verification; • Computing methodologies → Artificial intelligence; • Theory of computation → Constraint and logic programming.

show abstract

“…For instance, Meng et al [33] proposed an approach, namely TRANSFER, which leverages the deep semantic features and transferred knowledge from open-source data to improve fault localization. Wardat et al [34] proposed a debugging approach, namely DeepDiagnosis, which localizes the faults for DNN programs. Eniser et al [35] applied spectrum-based fault localization techniques to systematically identify suspicious neurons and then uses these neurons to synthesize new inputs, which is used for DNN testing and verification.…”

Section: Fault Localization In Deep Neural Networkmentioning

confidence: 99%

An Approach of Improving the Efficiency of Software Fault Localization based on Feedback Ranking Information

Yang,

Ma,

Guo

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Fault localization, a critical process of software debugging, can be implemented by ranking program statements according to their suspiciousness of being faulty, which, in turn, is calculated based on the execution behaviors of test cases. The performance of fault localization will deteriorate if the actual faulty statement is ranked low in suspiciousness. Intuitively speaking, the quality of the used test cases affects the suspiciousness ranking and thus the efficacy of fault localization. As such, it is necessary to generate test cases with “better” quality to improve the chance of faulty statements being ranked as highly suspicious. In this paper, we propose a software fault localization approach based on feedback ranking information, namely FLFR, according to an improved genetic algorithm. The starting point of the new method is the execution of a set of test cases, which gives a preliminary suspiciousness ranking of program statements. The improved genetic algorithm is iteratively applied to generate new test cases. The new method is evaluated through a series of experiments on four C programs and two Java programs. The experimental results show that the test cases automatically generated by the method can improve the suspiciousness ranking of the faulty statement, and thus enhance the performance of fault localization.

show abstract

DeepDiagnosis

Cited by 22 publications

References 16 publications

Bugs in machine learning-based systems: a faultload benchmark

Bugs in machine learning-based systems: a faultload benchmark

Semantic-Based Neural Network Repair

An Approach of Improving the Efficiency of Software Fault Localization based on Feedback Ranking Information

Contact Info

Product

Resources

About