A comprehensive study on challenges in deploying deep learning based software

Abstract

Deep learning (DL) becomes increasingly pervasive, being used in a wide range of software applications. These software applications, named as DL based software (in short as DL software), integrate DL models trained using a large data corpus with DL programs written based on DL frameworks such as TensorFlow and Keras. A DL program encodes the network structure of a desirable DL model and the process by which the model is trained using the training data. To help developers of DL software meet the new challenges posed by DL, enormous research e orts in software engineering have been devoted. Existing studies focus on the development of DL software and extensively analyze faults in DL programs. However, the deployment of DL software has not been comprehensively studied. To ll this knowledge gap, this paper presents a comprehensive study on understanding challenges in deploying DL software. We mine and analyze 3,023 relevant posts from Stack Over ow, a popular Q&A website for developers, and show the increasing popularity and high di culty of DL software deployment among developers. We build a taxonomy of speci c challenges encountered by developers in the process of DL software deployment through manual inspection of 769 sampled posts and report a series of actionable implications for researchers, developers, and DL framework vendors. CCS CONCEPTS • Software and its engineering → Software creation and management; • Computing methodologies → Arti cial intelligence; • General and reference → Empirical studies.

An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applications

“…is a general process for different types of DL based software applications [13]. First, developers construct structures of desirable DL models and specify run-time configuration (e.g., hyper-parameters) with DL programs written based on stateof-the-art DL frameworks such as TF and Keras.…”

“…In fact, developm ent o f D L m odels is a general process for different types of DL based applications [13] and its challenges Corresponding author: Xuanzhe Liu (xzl@pku.edu.cn).…”

“…For example, computation-intensive DL models can be executed efficiently on PC/server platforms, but they cannot be directly deployed and executed on mobile devices with limited computing power [23]. Although major vendors have rolled out specific DL frameworks such as TF Lite [24] and Core ML [25] to facilitate this deployment process, various specific faults are still emerging in this process and frequently asked on Stack Overflow (SO), one of the most popular Q&A forums for developers [13]. Moreover, previous work [13] has demonstrated that relevant questions are increasing rapidly on SO and more difficult to resolve than those related to other aspects of DL based applications.…”

“…Although major vendors have rolled out specific DL frameworks such as TF Lite [24] and Core ML [25] to facilitate this deployment process, various specific faults are still emerging in this process and frequently asked on Stack Overflow (SO), one of the most popular Q&A forums for developers [13]. Moreover, previous work [13] has demonstrated that relevant questions are increasing rapidly on SO and more difficult to resolve than those related to other aspects of DL based applications. In addition, mobile DL apps are not only used by billions of end users for their daily activities (e.g., speech-to-text and photo beauty) [2 2 ], [26], but also reported to be increasingly adopted in various safety-critical scenarios (e.g., driver assistance [27] and autonomous vehicles [28]).…”

“…We focus our study on the faults that occur during the usage of two representative frameworks specifically designed for deploying DL models on mobile devices, i.e., TF Lite [24] and Core ML [25], both of which are widely used in industry practice and well adopted in related studies [13], [23]. Specifically, we collect a dataset of 304 deployment faults related to their usage from SO and GitHub, two commonly used data sources for studying software faults [16]- [19], [29], By manual analysis, we qualitatively extract the symptom of each identified fault and construct a hierarchical taxonomy containing 23 symptom categories, indicating the diversity of deployment faults of mobile DL apps.…”