A Comparative Measurement Study of Deep Learning as a Service Framework

Abstract

Big data powered Deep Learning (DL) and its applications have blossomed in recent years, fueled by three technological trends: a large amount of digitized data openly accessible, a growing number of DL software frameworks in open source and commercial markets, and a selection of affordable parallel computing hardware devices. However, no single DL framework, to date, dominates in terms of performance and accuracy even for baseline classification tasks on standard datasets, making the selection of a DL framework an overwhelming task. This paper takes a holistic approach to conduct empirical comparison and analysis of four representative DL frameworks with three unique contributions. First, given a selection of CPU-GPU configurations, we show that for a specific DL framework, different configurations of its hyper-parameters may have a significant impact on both performance and accuracy of DL applications. Second, to the best of our knowledge, this study is the first to identify the opportunities for improving the training time performance and the accuracy of DL frameworks by configuring parallel computing libraries and tuning individual and multiple hyper-parameters. Third, we also conduct a comparative measurement study on the resource consumption patterns of four DL frameworks and their performance and accuracy implications, including CPU and memory usage, and their correlations to varying settings of hyper-parameters under different configuration combinations of hardware, parallel computing libraries. We argue that this measurement study provides in-depth empirical comparison and analysis of four representative DL frameworks, and offers practical guidance for service providers to deploying and delivering DL as a Service (DLaaS) and for application developers and DLaaS consumers to select the right DL frameworks for the right DL workloads.

Technical and imaging factors influencing performance of deep learning systems for diabetic retinopathy

“…This is with the exception of an Ensemble of multiple networks which often demonstrated superior results 24,36 . Previous studies examining different computational frameworks in the accuracy at general image classification tasks also showed comparable performance 37,38 . Although there have not been specific studies addressing the effect of compression of retinal images in the context of DL algorithms detection of DR, our study reinforces previous studies that have demonstrated the robustness of DL models with compression of general non-medical images up to a compression threshold 23 .…”

Denoising and Verification Cross-Layer Ensemble Against Black-box Adversarial Attacks

“…), or the hyperparameter settings (e.g., λ, training epochs, and random seed for weight initialization), because different ways of generating denoisers can have different effects with respect to the manifold and the convergence of the DNN learning [21]- [23]. The third approach to generate different denoisers is to use different optimization objectives, such as changing the distance function d from a simple perpixel loss to a perceptual loss [24], or using an advanced regularization such as a sparsity constraint [25].…”

“…First, we construct a set of base models as the verifiers for examining and repairing the target model prediction output, each is trained on the same dataset for the same task as the target model. Several techniques can be used to produce the base candidate models, such as varying neural network structures [23], training hyperparameters, or performing data augmentation [29]. One can also conduct snapshot learning [22] to obtain a set of model verifiers efficiently in a single run of model training.…”

Microscopy cell nuclei segmentation with enhanced U-Net

“…), the prediction results of deep model can be quite difficult to reproduce. To make things worse, implementations of deep models with different DL frameworks can also induce the variations in accuracy, running time [17,18] etc. In other words, one prerequisite condition for impartial model comparisons is to employ the same DL framework with comparative training settings and scheme.…”