Improving Performance of Data Science Applications in Python

Abstract

Objectives: The objective of this paper is to focus on the need of performance optimization of Data Science applications with an aim to enhance the speed, responsiveness, and effective resource utilization. Methods: This paper identifies the factors which effect the performance of an application and then classifies them into internal and external factors. This study mainly focuses on internal factors such as execution speed and space requirement. It uses code developed in Python as use cases to explore and examine features which improve the performance and effectiveness of Data Science (DS) applications. Findings: This paper provides comprehensive way of improving performance of applications by implementing features of Python like generator, vectorization, profiling, concurrency & parallelism, and caching. It is observed that space requirement reduces with the use of generators in code, leading to better space management. The conventional way of writing code in Python required additional 5.2 MiB (Mebibyte) space, while there was no additional space requirement with the use of generators in the same code. It is further observed that vectorization feature of Python helps in reducing execution time in comparison to the code written without vectorization. This paper further uses parallelism in the code and gets approximately 68% improvement in execution time. The implementation of profiling feature helps in identifying the time and space requirement which can be improved using features like generator, vectorization, concurrency, and parallelism. Other features like handling I/O operations, use of caching and choice between flat file data and database are also deliberated to improve the performance of code. Novelty: This paper highlights the potential of improving performance of DS applications using generator, vectorization, concurrency and parallelism features of Python and examines the effectiveness of the claim through its implementation. Keywords: Data Science, Generator, Vectorization, Parallelism, Profiling