Evolution of technical debt remediation in Python: A case study on the Apache Software Ecosystem

Abstract

In recent years, the evolution of software ecosystems and the detection of technical debt received significant attention by researchers from both industry and academia. While a few studies that analyze various aspects of technical debt evolution already exist, to the best of our knowledge, there is no large‐scale study that focuses on the remediation of technical debt over time in Python projects—that is, one of the most popular programming languages at the moment. In this paper, we analyze the evolution of technical debt in 44 Python open‐source software projects belonging to the Apache Software Foundation. We focus on the type and amount of technical debt that is paid back. The study required the mining of over 60K commits, detailed code analysis on 3.7K system versions, and the analysis of almost 43K fixed issues. The findings show that most of the repayment effort goes into testing, documentation, complexity, and duplication removal. Moreover, more than half of the Python technical debt is short term being repaid in less than 2 months. In particular, the observations that a minority of rules account for the majority of issues fixed and spent effort suggest that addressing those kinds of debt in the future is important for research and practice.

Technical Debt Repayment in Practice

“…Table 1.2 shows the research questions and the chapters in which they are addressed. Chapter 2 is based on a published article in the Journal of Software: Evolution and Process (JSEP) (Tan, Feitosa, Avgeriou, and Lungu 2021). In this paper, we analyzed the evolution of technical debt in 44 Python projects belonging to the Apache Software Foundation, including the type and amount of TD that was paid back.…”

“…Chapter 4 is based on a paper published in the 2020 International Conference on Technical Debt (TechDebt) (Tan, Feitosa and Avgeriou 2020) and its extension, published in the Information Software Technology (IST) (Tan, Feitosa and Avgeriou 2021). These two papers are, to the best of our knowledge, the first studies to exclusively focus on self-fixed technical debt.…”

“…Chapter 5 is based on a paper published in the 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) (Tan, Feitosa and Avgeriou 2021). In this paper, we conducted a survey with 17 well-known Java and Python open-source software communities to examine the phenomenon of self-fixed TD from the perspective of practitioners.…”

“…To handle this problem, we filter out the issues that disappeared due to file deletion. The same method was used in some previous studies (Digkas et al, 2018;Tan et al, 2021).…”

“…Recent studies have shown that a large percentage of TD is paid back during software evolution (Tan et al, 2021;Digkas et al, 2018;Lenarduzzi et al, 2019;Baldassarre et al, 2020). Tan et al (2020), established that a considerable amount of TD items are repaid by the same developers who introduced them.…”