The Prevalence of Code Smells in Machine Learning projects

Abstract

Artificial Intelligence (AI) and Machine Learning (ML) are pervasive in the current computer science landscape. Yet, there still exists a lack of software engineering experience and best practices in this field. One such best practice, static code analysis, can be used to find code smells, i.e., (potential) defects in the source code, refactoring opportunities, and violations of common coding standards. Our research set out to discover the most prevalent code smells in ML projects. We gathered a dataset of 74 open-source ML projects, installed their dependencies and ran Pylint on them. This resulted in a top 20 of all detected code smells, per category. Manual analysis of these smells mainly showed that code duplication is widespread and that the PEP8 convention for identifier naming style may not always be applicable to ML code due to its resemblance with mathematical notation. More interestingly, however, we found several major obstructions to the maintainability and reproducibility of ML projects, primarily related to the dependency management of Python projects. We also found that Pylint cannot reliably check for correct usage of imported dependencies, including prominent ML libraries such as PyTorch.

"Project smells" -- Experiences in Analysing the Software Quality of ML Projects with mllint

“…Dependency Management Dependency management was done well in two projects, in one project not at all and in other projects with a combination of requirements.txt and setup.py, of which mllint doesn't recognise whether it is used in an effective, maintainable and reproducible way. Manual inspection showed that these projects do groom their requirements.txt files, there was no evidence of direct pip freeze usage as was prevalent in [22] and some of these projects were neatly separating their runtime dependencies from development dependencies. However, there were also two projects that duplicated the contents of their requirements.txt in their setup.py.…”

“…Dependency Management This category entails checking whether the project manages its code dependencies (e.g. used libraries) in a reproducible and maintainable manner, to mitigate the dependency management issues found in [22]. Continuous Integration The rule in this category checks whether the project has a CI configuration file.…”

“…Especially in ML projects, code smells are only a small piece in the software quality puzzle. We noticed this first-hand in our previous research on the prevalence in code smells in open-source ML projects: nearly half of the analysed projects struggled with managing their code dependencies [22]. We realised that a more holistic approach to code smells, 'project smells', would be required.…”

“…Thus, to more accurately assess the software quality of ML applications, a more holistic approach would be required, where instead of code smells, we analyse project smells. Such project smells are concerned with deficits in how an ML project is managed, including poor dependency management (as outlined in [22]), lack of version control for code or data, unit testing, proper Continuous Integration (CI) configurations, or effective static analysis tooling. Code smells are also a subcategory of project smells.…”

“…Lastly, in our previous work, we analysed the prevalence of code smells in ML projects [22]. Aside from widespread code duplication in ML projects and several false positives in Pylint, we coincidentally found that nearly half of the projects we analysed struggled with dependency management, so much so that manual adjustments were needed to allow error-free installation of the Python libraries that they used.…”