Addressing problems with replicability and validity of repository mining studies through a smart data platform

Trautsch, Fabian; Herbold, Steffen; Makedonski, Philip; Grabowski, Jens

doi:10.1007/s10664-017-9537-x

Cited by 42 publications

(42 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used SmartSHARK [39] to collect data from a convenience sample of fifteen Apache projects to conduct our experiments. 2 .…”

Section: Datamentioning

confidence: 99%

On the Costs and Profit of Software Defect Prediction

Herbold

2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Defect prediction can be a powerful tool to guide the use of quality assurance resources. However, while lots of research covered methods for defect prediction as well as methodological aspects of defect prediction research, the actual cost saving potential of defect prediction is still unclear. Within this article, we close this research gap and formulate a cost model for software defect prediction. We derive mathematically provable boundary conditions that must be fulfilled by defect prediction models such that there is a positive profit when the defect prediction model is used. Our cost model includes aspects like the costs for quality assurance, the costs of post-release defects, the possibility that quality assurance fails to reveal predicted defects, and the relationship between software artifacts and defects. We initialize the cost model using different assumptions, perform experiments to show trends of the behavior of costs on real projects. Our results show that the unrealistic assumption that defects only affect a single software artifact, which is a standard practice in the defect prediction literature, leads to inaccurate cost estimations. Moreover, the results indicate that thresholds for machine learning metrics are also not suited to define success criteria for software defect prediction.Index Terms-Defect prediction, costs, return on investment ! • S. Herbold is with the

show abstract

“…We used SmartSHARK [39] to collect data from a convenience sample of fifteen Apache projects to conduct our experiments. 2 .…”

Section: Datamentioning

confidence: 99%

On the Costs and Profit of Software Defect Prediction

Herbold

2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Over the past couple of years, multiple tools and techniques are introduced to study software evolution by commit‐level . Some of them do not run analysis on source code .…”

Section: Squaad Toolsetmentioning

confidence: 99%

“…Consequently, the execution of the study requires strong on‐premise resource and takes multiple weeks . There are some mining techniques designed to scale by running static analysis on different revisions of each file in parallel . These techniques are extremely efficient in generating an Abstract Syntax Tree (AST) and calculating file‐based quality metrics, such as size, cohesion, and complexity .…”

Section: Squaad Toolsetmentioning

confidence: 99%

“…There are some mining techniques designed to scale by running static analysis on different revisions of each file in parallel . These techniques are extremely efficient in generating an Abstract Syntax Tree (AST) and calculating file‐based quality metrics, such as size, cohesion, and complexity . They can also aggregate the result of individual files analysis to evaluate the quality of a project.…”

Section: Squaad Toolsetmentioning

confidence: 99%

“…As soon as the framework is configured for a subject system, we can run the represented analysis on that system and study its evolution. This full automation makes analysis conducted by SQUAAD replicable and addresses a major threat (ie, not being replicable) to the external validity of repository mining studies . We have also developed web interfaces to illustrate the evolution of different quality attributes and the impact of each developer on software quality.…”

Section: Squaad Toolsetmentioning

confidence: 99%

See 2 more Smart Citations

Anticipatory development processes for reducing total ownership costs and schedules

Boehm

Behnamghader

2019

Systems Engineering

View full text Add to dashboard Cite

Many systems and software processes overfocus on getting a project and product from an initial set of requirements to an Initial Operational Capability (IOC). Examples are most waterfall and V models. Projects following such processes may pass acceptance tests for functionality and performance, but may leave the product with serious maintainability shortfalls. Many agile processes focus on users' initial usage priorities, but often make development commitments for earlier needs that are incompatible with achieving later critical needs (eg, security, safety). Incremental development process models can do better, but often later increments may find that the earlier increments have not prepared them for ease of modification and repair. Besides increasing Total Ownership Costs (TOCs), long mean times to repair result in long downtimes, which can be critical to an organization's income and reputation. Furthermore, many of these shortfalls take the form of Technical Debt (TD), in that the later they are fixed, the more slow and expensive will be the fixes. This paper summarizes three process frameworks and tools providing more anticipatory ways to improve systems and software maintainability and life‐cycle cost‐effectiveness. The first framework is an Opportunity Tree for identifying and anticipating such ways. The second framework, Software Quality Understanding by Analysis of Abundant Data (SQUAAD), is a toolset for tracking a software project's incremental code commits, and analyzing and visualizing each commit's incremental and cumulative TD. The third framework is a Software/Systems Maintenance Readiness Framework (SMRF), that identifies needed maintenance readiness levels at development decision reviews, similar to the Technology Readiness Levels (TRLs) framework.

show abstract

An ensemble‐based predictive mutation testing approach that considers impact of unreached mutants

Aghamohammadi

Mirian-Hosseinabadi

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

Predictive mutation testing (PMT) is a technique to predict whether a mutant is killed, using machine learning approaches. Researchers have proposed various methods for PMT over the years. However, the impact of unreached mutants on PMT is not fully addressed. A mutant is unreached if the statement on which the mutant is generated is not executed by any test cases. We aim at showing that unreached mutants can inflate PMT results. Moreover, we propose an alternative approach to PMT, suggesting a different interpretation for PMT. To this end, we replicated the previous PMT research. We empirically evaluated the suggested approach on 654 Java projects provided by prior literature. Our results indicate that the performance of PMT drastically decreases in terms of area under a receiver operating characteristic curve (AUC) from 0.833 to 0.517. Furthermore, PMT performs worse than random guesses on 27% of the projects. The proposed approach improves the PMT results, achieving the average AUC value of 0.613. As a result, we recommend researchers to remove unreached mutants when reporting the results.

show abstract

Addressing problems with replicability and validity of repository mining studies through a smart data platform

Cited by 42 publications

References 21 publications

On the Costs and Profit of Software Defect Prediction

On the Costs and Profit of Software Defect Prediction

Anticipatory development processes for reducing total ownership costs and schedules

An ensemble‐based predictive mutation testing approach that considers impact of unreached mutants

Contact Info

Product

Resources

About