Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

Tambon, Florian; Nikanjam, Amin; An, Lê Văn; Khomh, Foutse; Antoniol, Giuliano

doi:10.48550/arxiv.2112.13314

Cited by 3 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their studies reveal that the root causes of a large portion of reported bugs reside in the algorithm implementations or the interfaces (i.e., API) provided by TensorFlow. Tambon et al [50] study silent bugs such as performance or accuracy issues inside TensorFlow. Silent bugs will lead to incorrect behavior, but they do not cause system crashes or hang, nor show an error message to users.…”

Section: Empirical Studies For DL Librariesmentioning

confidence: 99%

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

Meiziniu

Cao

Tian

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Recent deep learning (DL) applications are mostly built on top of DL libraries. The quality assurance of these libraries is critical to the dependable deployment of DL applications. Techniques have been proposed to generate various DL models and apply them to test these libraries. However, their test effectiveness is constrained by the diversity of layer API calls in their generated DL models. Our study reveals that these techniques can cover at most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer sequences. As a result, we find that many bugs arising from specific layer API calls (i.e., specific layer inputs, parameter values, or layer sequences) can be missed by existing techniques. Because of this limitation, we propose COMET to effectively generate DL models with diverse layer API calls for DL library testing. COMET: (1) designs a set of mutation operators and a coverage-based search algorithm to diversify layer inputs, layer parameter values, and layer sequences in DL models. (2) proposes a model synthesis method to boost the test efficiency without compromising the layer API call diversity. Our evaluation result shows that COMET outperforms baselines by covering twice as many layer inputs (69.7% vs. 34.1%), layer parameter values (50.2% vs. 25.9%), and layer sequences (39.0% vs. 15.6%) as those by the state-of-the-art. Moreover, COMET covers 3.4% more library branches than those by existing techniques. Finally, COMET detects 32 new bugs in the latest version of eight popular DL libraries, including TensorFlow and MXNet, with 21 of them confirmed by DL library developers and 7 of those confirmed bugs have been fixed by developers.

show abstract

Section: Empirical Studies For DL Librariesmentioning

confidence: 99%

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

Meiziniu

Cao

Tian

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

show abstract

“…Liu et al [26] and Guo et ai [16] propose the use of inconsistency patterns for testing across different DL libraries. Tambon et al [36] studied a typical bug in DL library, i.e., silent bugs. The silent bugs refer to bugs that lead to wrong behavior, but no apparent bug symptoms (e.g., system crash, hang, or exceptions raised with error messages).…”

Section: Empirical Study For DL Librariesmentioning

confidence: 99%

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

Meiziniu¹,

Cao²,

Tian³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent deep learning (DL) applications are mostly built on top of DL libraries. The quality assurance of these libraries is critical to the dependable deployment of DL applications. A few techniques have thereby been proposed to test DL libraries by generating DL models as test inputs. Then these techniques feed those DL models to DL libraries for making inferences, in order to exercise DL libraries modules related to a DL model's execution. However, the test effectiveness of these techniques is constrained by the diversity of generated DL models. Our investigation finds that these techniques can cover at most 11.7% of layer pairs (i.e., call sequence between two layer APIs) and 55.8% of layer parameters (e.g., "padding" in Conv2D). As a result, we find that many bugs arising from specific layer pairs and parameters can be missed by existing techniques.In view of the limitations of existing DL library testing techniques, we propose MEMO to efficiently generate diverse DL models by exploring layer types, layer pairs, and layer parameters. MEMO:(1) designs an initial model reduction technique to boost test efficiency without compromising model diversity; and (2) designs a set of mutation operators for a customized Markov Chain Monte Carlo (MCMC) algorithm to explore new layer types, layer pairs, and layer parameters. We evaluate MEMO on seven popular DL libraries, including four for model execution (TensorFlow, PyTorch and MXNet, and ONNX) and three for model conversions (Keras-MXNet, TF2ONNX, ONNX2PyTorch). The evaluation result shows that MEMO outperforms recent works by covering 10.3% more layer pairs, 15.3% more layer parameters, and 2.3% library branches. Moreover, MEMO detects 29 new bugs in the latest version of DL libraries, with 17 of them confirmed by DL library developers, and 5 of those confirmed bugs have been fixed.

show abstract

“…Bad performance refers to the consequence of a bug where the accuracy of the trained model is negatively affected. The severity of such bugs is particularly high if these bugs occur "silently", i.e., without the user noticing it [27].…”

Section: Bugs In DL Softwarementioning

confidence: 99%

“…The subcategory of bugs named "silent bugs" has been studied by Tambon et al [27]. These bugs lead to the wrong behavior of the system, but they do not cause crashes or hangs, nor do they indicate any error message to the user.…”

Section: Bugs In DL Softwarementioning

confidence: 99%

See 1 more Smart Citation

How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models?

Shahriari

Ramler

Fischer

2022

MAKE

View full text Add to dashboard Cite

In the last decade, industry’s demand for deep learning (DL) has increased due to its high performance in complex scenarios. Due to the DL method’s complexity, experts and non-experts rely on blackbox software packages such as Tensorflow and Pytorch. The frameworks are constantly improving, and new versions are released frequently. As a natural process in software development, the released versions contain improvements/changes in the methods and their implementation. Moreover, versions may be bug-polluted, leading to the model performance decreasing or stopping the model from working. The aforementioned changes in implementation can lead to variance in obtained results. This work investigates the effect of implementation changes in different major releases of these frameworks on the model performance. We perform our study using a variety of standard datasets. Our study shows that users should consider that changing the framework version can affect the model performance. Moreover, they should consider the possibility of a bug-polluted version before starting to debug source code that had an excellent performance before a version change. This also shows the importance of using virtual environments, such as Docker, when delivering a software product to clients.

show abstract

Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

Cited by 3 publications

References 15 publications

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models?

Contact Info

Product

Resources

About