An Empirical Study of Common Challenges in Developing Deep Learning Applications

Zhang, Tianyi; Gao, Cuiyun; Ма, Лей; Lyu, Michael R.; Kim, Miryung

doi:10.1109/issre.2019.00020

Cited by 108 publications

(78 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [40] studied 715 Stack Overflow bug related posts for TensorFlow, PyTorch, and Deeplearning4j to classify the questions into 7 different categories and built an automated tool that categorizes questions based on the frequently found words from each category and computing the tf-idf value with respect to the keywords. Also, the authors have studied the challenges of answering the question in Stack Overflow by calculating the response time for each category and have found 5 categories of root causes for the bug related posts.…”

Section: Related Workmentioning

confidence: 99%

Repairing deep neural networks

Islam

Pan

Nguyen

et al. 2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

View full text Add to dashboard Cite

Significant interest in applying Deep Neural Network (DNN) has fueled the need to support engineering of software that uses DNNs. Repairing software that uses DNNs is one such unmistakable SE need where automated tools could be beneficial; however, we do not fully understand challenges to repairing and patterns that are utilized when manually repairing DNNs. What challenges should automated repair tools address? What are the repair patterns whose automation could help developers? Which repair patterns should be assigned a higher priority for building automated bug repair tools? This work presents a comprehensive study of bug fix patterns to address these questions. We have studied 415 repairs from Stack Overflow and 555 repairs from GitHub for five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand challenges in repairs and bug repair patterns. Our key findings reveal that DNN bug fix patterns are distinctive compared to traditional bug fix patterns; the most common bug fix patterns are fixing data dimension and neural network connectivity; DNN bug fixes have the potential to introduce adversarial vulnerabilities; DNN bug fixes frequently introduce new bugs; and DNN bug localization, reuse of trained model, and coping with frequent releases are major challenges faced by developers when fixing bugs. We also contribute a benchmark of 667 DNN (bug, repair) instances. CCS CONCEPTS • Software and its engineering → Software defect analysis; • Computing methodologies → Machine learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Repairing deep neural networks

Islam

Pan

Nguyen

et al. 2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

View full text Add to dashboard Cite

show abstract

“…It has been a common practice for SE researchers to get insight into developers' concerns on different SE issues by mining related posts from SO [5,7,8,23,30,33]. In our study, we use SO as the data source because: (i) as one of the most popular community-driven Q&A websites, the users in SO range from novices to experts, increasing the diversity of the analyzed issues; (ii) developers often seek for help in SO after they cannot find solutions in documents or internet search, leading to more unsolvable and non-trivial build problems in our analyzed data; (iii) SO inherently contains build issues with implicit symptoms which are often hard to be captured in reproduced or historical build data, increasing comprehensiveness of the dataset.…”

Section: Data Collectionmentioning

confidence: 99%

“…A major threat to validity is that we only use Stack Overflow as the data source to study how developers resolve build issues. Although the study is based on a representative sample of SO posts and SO posts have been widely used in previous work [5,7,8,10,23,30,33], there could be build issues that are never discussed on SO. In other words, we cannot guarantee the generalizability of our observations due to the bias induced by single data source.…”

Section: Threats To Validitymentioning

confidence: 99%

Understanding build issue resolution in practice: symptoms and fix patterns

Lou

Chen

Cao

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

Build systems are essential for modern software maintenance and development, while build failures occur frequently across software systems, inducing non-negligible costs in development activities. Build failure resolution is a challenging problem and multiple studies have demonstrated that developers spend non-trivial time in resolving encountered build failures; to relieve manual efforts, automated resolution techniques are emerging recently, which are promising but still limitedly effective. Understanding how build failures are resolved in practice can provide guidelines for both developers and researchers on build issue resolution. Therefore, this work presents a comprehensive study of fix patterns in practical build failures. Specifically, we study 1,080 build issues of three popular build systems Maven, Ant, and Gradle from Stack Overflow, construct a fine-granularity taxonomy of 50 categories regarding to the failure symptoms, and summarize the fix patterns for different failure types. Our key findings reveal that build issues stretch over a wide spectrum of symptoms; 67.96% of the build issues are fixed by modifying the build script code related to plugins and dependencies; and there are 20 symptom categories, more than half of whose build issues can be fixed by specific patterns. Furthermore, we also address the challenges in applying non-intuitive or simplistic fix patterns for developers. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

show abstract

“…The increasing dependence of current software applications on DL (as in DL software) makes it a crucial topic in the software engineering (SE) research community. Speci cally, many research e orts [80,84,85,106,108] have been devoted to characterizing the new challenges that DL poses to software development. To characterize the challenges that developers encounter in this process, various studies [85,106,108] focus on analyzing faults in DL programs.…”

Section: Introductionmentioning

confidence: 99%

“…Speci cally, many research e orts [80,84,85,106,108] have been devoted to characterizing the new challenges that DL poses to software development. To characterize the challenges that developers encounter in this process, various studies [85,106,108] focus on analyzing faults in DL programs. For instance, Islam et al [85] have presented a comprehensive study of faults in DL programs written based on TensorFlow (TF) [70], Keras [62], PyTorch [63], Theano [73], and Ca e [86] frameworks.…”

Section: Introductionmentioning

confidence: 99%

A comprehensive study on challenges in deploying deep learning based software

Chen

Cao

Liu

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

Deep learning (DL) becomes increasingly pervasive, being used in a wide range of software applications. These software applications, named as DL based software (in short as DL software), integrate DL models trained using a large data corpus with DL programs written based on DL frameworks such as TensorFlow and Keras. A DL program encodes the network structure of a desirable DL model and the process by which the model is trained using the training data. To help developers of DL software meet the new challenges posed by DL, enormous research e orts in software engineering have been devoted. Existing studies focus on the development of DL software and extensively analyze faults in DL programs. However, the deployment of DL software has not been comprehensively studied. To ll this knowledge gap, this paper presents a comprehensive study on understanding challenges in deploying DL software. We mine and analyze 3,023 relevant posts from Stack Over ow, a popular Q&A website for developers, and show the increasing popularity and high di culty of DL software deployment among developers. We build a taxonomy of speci c challenges encountered by developers in the process of DL software deployment through manual inspection of 769 sampled posts and report a series of actionable implications for researchers, developers, and DL framework vendors. CCS CONCEPTS • Software and its engineering → Software creation and management; • Computing methodologies → Arti cial intelligence; • General and reference → Empirical studies.

show abstract

An Empirical Study of Common Challenges in Developing Deep Learning Applications

Cited by 108 publications

References 37 publications

Repairing deep neural networks

Repairing deep neural networks

Understanding build issue resolution in practice: symptoms and fix patterns

A comprehensive study on challenges in deploying deep learning based software

Contact Info

Product

Resources

About