Apricot: A Weight-Adaptation Approach to Fixing Deep Learning Models

Zhang, Hao; Chan, W. K.

doi:10.1109/ase.2019.00043

Cited by 65 publications

(40 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [75] introduced an automatic approach to fixing deep learning models called Apricot. Apricot is able to adjust the ill-trained weights without using additional training data or any artificial parameters, Apricot using a set of reduced models from the original model, and compare the differences between the original model and correct/incorrect of reduced models iteratively, to find these failing test case that are responsible for the misclassification in the original model.…”

Section: Bugs Repairing In Deep Learningmentioning

confidence: 99%

See 1 more Smart Citation

DeepLocalize: Fault Localization for Deep Neural Networks

Wardat

Rajan

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Deep neural networks (DNNs) are becoming an integral part of most software systems. Previous work has shown that DNNs have bugs. Unfortunately, existing debugging techniques don t support localizing DNN bugs because of the lack of understanding of model behaviors. The entire DNN model appears as a black box.To address these problems, we propose an approach and a tool that automatically determines whether the model is buggy or not, and identifies the root causes for DNN errors. Our key insight is that historic trends in values propagated between layers can be analyzed to identify faults, and also localize faults. To that end, we first enable dynamic analysis of deep learning applications: by converting it into an imperative representation and alternatively using a callback mechanism. Both mechanisms allows us to insert probes that enable dynamic analysis over the traces produced by the DNN while it is being trained on the training data. We then conduct dynamic analysis over the traces to identify the faulty layer or hyperparameter that causes the error. We propose an algorithm for identifying root causes by capturing any numerical error and monitoring the model during training and finding the relevance of every layer/parameter on the DNN outcome. We have collected a benchmark containing 40 buggy models and patches that contain real errors in deep learning applications from Stack Overflow and GitHub. Our benchmark can be used to evaluate automated debugging tools and repair techniques. We have evaluated our approach using this DNN bug-and-patch benchmark, and the results showed that our approach is much more effective than the existing debugging approach used in the state-of-the-practice Kerns library. For 34/40 cases, our approach was able to detect faults whereas the best debugging approach provided by Kerns detected 32/40 faults. Our approach was able to localize 21/40 bugs whereas Keras did not localize any faults.

show abstract

Section: Bugs Repairing In Deep Learningmentioning

confidence: 99%

“…In recent years, several researchers are supporting automated debugging and repair approaches for deep neural networks, and recent research is summarized in [72]. This topic is still at the early stages [74], [75]. To the best of our knowledge, all previous works are focused on the training bugs.…”

Section: Bugs Repairing In Deep Learningmentioning

confidence: 99%

DeepLocalize: Fault Localization for Deep Neural Networks

Wardat

Rajan

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

show abstract

“…We ran MODE on the MNIST models from our study. The results are as follows: Unlike MODE that identifies ill-trained weights or buggy neurons, Apricot [24] first generates a set of models from the original neural network with a reduced set of training data and at each iteration of the training, Apricot adjusts each weight of the repaired model towards the average weight of these reduced models correctly classifying the input while away from the misclassifications. The approach from [19] uses constraint solving for repairing neural networks.…”

Section: Related Workmentioning

confidence: 99%

NNrepair: Constraint-Based Repair of Neural Network Classifiers

Usman

Gopinath

Sun

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We present NNrepair, a constraint-based technique for repairing neural network classifiers. The technique aims to fix the logic of the network at an intermediate layer or at the last layer. NNrepair first uses fault localization to find potentially faulty network parameters (such as the weights) and then performs repair using constraint solving to apply small modifications to the parameters to remedy the defects. We present novel strategies to enable precise yet efficient repair such as inferring correctness specifications to act as oracles for intermediate layer repair, and generation of experts for each class. We demonstrate the technique in the context of three different scenarios: (1) Improving the overall accuracy of a model, (2) Fixing security vulnerabilities caused by poisoning of training data and (3) Improving the robustness of the network against adversarial attacks. Our evaluation on MNIST and CIFAR-10 models shows that NNrepair can improve the accuracy by 45.56% points on poisoned data and 10.40% points on adversarial data. NNrepair also provides small improvement in the overall accuracy of models, without requiring new data or re-training.

show abstract

“…Some research studies common mistakes in programs that design and train neural networks [16]- [19] or other types of machine learning models (e.g., SVM and decision tree) [93]. Some works focus on testing [20]- [45] and fixing [46]- [49] neural networks. All of these studies consider building machine learning models, instead of using them.…”

Section: Re L a T E D Wo R Kmentioning

confidence: 99%

Are Machine Learning Cloud APIs Used Correctly?

Wan

Liu

Hoffmann

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

University o f Chicago {cw an, s h ic h e n g 2 0 0 0 , h a n k h o ffm a n n , m m a ire , s h a n lu } @ u c h ic a g o .e d u Abstract-Machine learning (ML) cloudAPIs enable developers to easily incorporate learning solutions into software systems. Unfortunately, ML APIs are challenging to use correctly and efficiently, given their unique semantics, data requirements, and accuracy-performance tradeoffs. Much prior work has studied how to develop ML APIs or ML cloud services, but not how open-source applications are using ML APIs. In this paper, we manually studied 360 representative open-source applications that use Google or AWS cloud-based m L APIs, and found 70% of these applications contain API misuses in their latest versions that degrade functional, performance, or economical quality of the software. We have generalized 8 anti-patterns based on our manual study and developed automated checkers that identify hundreds of more applications that contain ML API misuses. I. In t r o d u c t io n A. MotivationMachine learning (ML) provides efficient solutions for a number of problems that were difficult to solve with traditional computing techniques; e.g., object detection and language translation. ML cloud APIs allow programmers to incorporate these learning solutions into software systems without designing and training the learning model themselves [1], and hence put these powerful techniques into the hands of non-experts. Indeed, there are more than 35,000 open-source projects on GitHub that use Google or Amazon ML Cloud APIs to solve a wide variety of problems, among which more than 14,000 were created within the last 1 2 months.While these APIs make it easy for non-experts to incorporate learning into software systems, there are still a number of challenges that must be addressed to ensure that the resulting applications are both correct and efficient. While certain challenges come with the use of any third-party API, this paper focuses on unique challenges for ML APIs that arise due to the nature of learning itself. Complicated data requirements. Machine learning tech-niques are used to process digitalized real-world visual, audio and text content. Although such content can be generated by a huge variety of devices and encoding software, the suitable input content and format (encoding, resolution, size, etc.) for ML APIs are rather limited and often uniquely defined by the DNN-training process. For example, cameras can produce images in many formats, but the image sets on which ML models are trained have a relatively small variety [2 ]- [8 ]. Thus, it is up to the API user to select the input or convert the input into what the API can accept and effectively process.

show abstract

Apricot: A Weight-Adaptation Approach to Fixing Deep Learning Models

Cited by 65 publications

References 27 publications

DeepLocalize: Fault Localization for Deep Neural Networks

DeepLocalize: Fault Localization for Deep Neural Networks

NNrepair: Constraint-Based Repair of Neural Network Classifiers

Are Machine Learning Cloud APIs Used Correctly?

Contact Info

Product

Resources

About