Prevalence, Contents and Automatic Detection of KL-SATD

Rantala, Leevi; Mäntylä, Martti; Lo, David

doi:10.1109/seaa51224.2020.00069

Cited by 7 publications

(8 citation statements)

References 14 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different approaches have been proposed to detect SATD, including a keyword-based (da S. Maldonado and Shihab, 2015), text mining (Huang et al, 2018), Natural Language Processing (da S. Maldonado et al, 2017b, and convolutional neural networks (Ren et al, 2019). Rantala et al (2020) went deeper on SATD comments containing keywords such as TODO and FIXME (i.e., KL-SATD) and compared them to the rest of source code comments. While the median percentage of KL-SATD is very low ( 2%), their contents is very different from other comments.…”

Section: Studies On Technical Debtmentioning

confidence: 99%

“…Examples of SATD by Potdar and Shihab (2014) are such comments as "// TODO this is such a hack it is silly" from Eclipse and "// Unsafe; should error" from Chromium OS. Following the seminal work of Potdar and Shihab (2014) researchers have studied different kinds of TD (Bavota and Russo, 2016;Fucci et al, 2021;Maipradit et al, 2020;Mensah et al, 2018;Rantala et al, 2020): from those introduced by such keywords as "TODO" or "FIXME" to those focusing on situations when developers are waiting for a certain event or an updated functionality, from those focusing on functional defects to those related to maintainability. To encompass different variants of SATD found in the scientific literature, in this paper we opt for a broad definition and consider as SATD any source code comments for annotating delayed or intended work activities such as TODO, FIXME, hack, workaround.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-admitted technical debt practices: a comparison between industry and open-source

et al. 2021

View full text Add to dashboard Cite

Self-admitted technical debt (SATD) consists of annotations, left by developers as comments in the source code or elsewhere, as a reminder about pieces of software manifesting technical debt (TD), i.e., "not being ready yet". While previous studies have investigated SATD management and its relationship with software quality, there is little understanding of the extent and circumstances to which developers admit TD. This paper reports the results of a study in which we asked developers from industry and opensource about their practices in annotating source code and other artifacts for self-admitting TD. The study consists of two phases. First, we conducted 10 interviews to gather a first understanding of the phenomenon and to prepare a survey questionnaire. Then, we surveyed 52 industrial developers as well as 49 contributors to open-source projects. Results of the study show how the TD annotation practices, as well as the typical content of SATD comments, are very similar between open-source and industry. At the same time, our results highlight how, while open-source code is spread of comments admitting the need for improvements, SATD in industry may be dictated by organizational guidelines but, at the same time, implicitly discouraged by the fear of admitting

show abstract

Section: Studies On Technical Debtmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Self-admitted technical debt practices: a comparison between industry and open-source

et al. 2021

View full text Add to dashboard Cite

show abstract

“…When investigating feature importance, in a single project one would suspect that project specific words would have more weight. Indeed this is a difficulty others have experienced when using regression coefficients as measures of feature importance [44]. It appears that SHAP avoids this problem as only one project specific word 'nonnls' is seen in the table.…”

Section: Satd Feature Interpretability and Analysismentioning

confidence: 99%

Data Balancing Improves Self-Admitted Technical Debt Detection

Sridharan

Mäntylä

Rantala

et al. 2021

2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

Self Cite

View full text Add to dashboard Cite

A high imbalance exists between technical debt and non-technical debt source code comments. Such imbalance affects Self Admitted Technical Debt (SATD) detection performance, and existing literature lacks empirical evidence on the choice of balancing technique. In this work, we evaluate the impact of multiple balancing techniques, including Data level, Classifier level, and Hybrid, for SATD detection in Within-Project and Cross-Project setup. Our results show that the Data level balancing technique SMOTE or Classifier level Ensemble approaches with Random Forest or XGBoost are reasonable choices depending on whether the goal is to maximize Precision, Recall, F1, or AUC-ROC. We compared our best-performing model with the previous SATD detection benchmark (cost-sensitive Convolution Neural Network). Interestingly the top-performing XGBoost with SMOTE sampling improved the Within-project F1 score by 10% but fell short in Cross-Project set up by 9%. This supports the higher generalization capability of deep learning in Crossproject SATD detection, yet while working within individual projects, classical machine learning algorithms can deliver better performance. We also evaluate and quantify the impact of duplicate source code comments in SATD detection performance. Finally, we employ SHAP and discuss the interpreted SATD features. We have included the replication package 1 and shared a web-based SATD prediction tool 2 with the balancing techniques in this study.

show abstract

“…Potdar and Shihab [75] explore self-admitted, e.g., via code comments, technical debt (SATD), while Bavota and Russo [76] investigate the diffusion and evolution of SATD and its relationship with software quality. Huang et al [77] and Rantala et al [78] identify SATD using advanced techniques, and Christians [79] examines the relation between SATD and refactoring in general systems. The refactorings we have identified that correlate to debt categories may be considered a form of (ML-specific) SATD.…”

Section: Re L a T E D W O R Kmentioning

confidence: 99%

An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems

Tang

Khatchadourian

Bagherzadeh

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Machine Learning (ML), including Deep Learning (DL), systems, i.e., those with ML capabilities, are pervasive in today's data-driven society. Such systems are complex; they are comprised of ML models and many subsystems that support learning processes. As with other complex systems, ML systems are prone to classic technical debt issues, especially when such systems are long-lived, but they also exhibit debt specific to these systems. Unfortunately, there is a gap of knowledge in how ML systems actually evolve and are maintained. In this paper, we fill this gap by studying refactorings, i.e., source-to-source semanticspreserving program transformations, performed in real-world, open-source software, and the technical debt issues they alleviate. We analyzed 26 projects, consisting of 4.2 MLOC, along with 327 manually examined code patches. The results indicate that developers refactor these systems for a variety of reasons, both specific and tangential to ML, some refactorings correspond to established technical debt categories, while others do not, and code duplication is a major crosscutting theme that particularly involved ML configuration and model code, which was also the most refactored. We also introduce 14 and 7 new ML-specific refactorings and technical debt categories, respectively, and put forth several recommendations, best practices, and anti-patterns. The results can potentially assist practitioners, tool developers, and educators in facilitating long-term ML system usefulness.Index Terms-empirical studies, refactoring, machine learning systems, technical debt, software repository mining I. In t r o d u c t io nIn the big data era, Machine Learning (ML), including Deep Learning (DL), systems are pervasive in modern society. Central to these systems are dynamic ML models, whose behavior is ultimately defined by their input data. However, such systems do not only consist of ML models; instead, ML systems typically encompass complex subsystems that support ML processes [1]. ML systems-like other long-lived, complex systems-are prone to classic technical debt [2 ] issues; yet, they also exhibit debt specific to such systems [3]. While work exist on applying software engineering (SE) rigor to ML systems [4]- [12], there is generally a gap of knowledge in how ML systems actually evolve and are maintained. As ML systems become more difficult and expensive to maintain [1 ], understanding the kinds of modifications developers are required to make to such systems-our overarching research question-is of the utmost importance.To fill this gap, we performed an empirical study on common refactorings, i.e., source-to-source semantics preserving program transformations-a widely accepted mechanism for effectively reducing technical debt [13]-[16]-in real-world, open-source ML systems. We set out to discover (i) the kinds of refactorings-both specific and tangential to ML-performed, (ii) whether particular refactorings occurred more often in model code vs. other supporting subsystems, (iii) the types of technical debt ...

show abstract

Prevalence, Contents and Automatic Detection of KL-SATD

Cited by 7 publications

References 14 publications

Self-admitted technical debt practices: a comparison between industry and open-source

Self-admitted technical debt practices: a comparison between industry and open-source

Data Balancing Improves Self-Admitted Technical Debt Detection

An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems

Contact Info

Product

Resources

About