Less is More: Temporal Fault Predictive Performance over Multiple Hadoop Releases

Harman, Mark; Islam, Syed; Jia, Yunyi; Minku, Leandro L.; Sarro, Federica; Srivisut, Komsan

doi:10.1007/978-3-319-09940-8_19

Cited by 34 publications

(33 citation statements)

References 17 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For every considered release, we iteratively train on the previous release(s) and evaluate on the current one. We consider two typical cases addressed in previous work: training on the last release [18,20] and training on the last three releases [37,38]. We start the evaluation from the fourth release onwards (as we need at least three releases on which to train the predictive models) and we consider releases with at least 10 vulnerable components.…”

Section: Experimental Design and Analysis 61 Methodologymentioning

confidence: 99%

The importance of accounting for real-world labelling when predicting software vulnerabilities

Jiménez

Rwemalika

Papadakis

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

Self Cite

View full text Add to dashboard Cite

Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mislead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings. CCS CONCEPTS • Software and its engineering → Software defect analysis.

show abstract

Section: Experimental Design and Analysis 61 Methodologymentioning

confidence: 99%

The importance of accounting for real-world labelling when predicting software vulnerabilities

Jiménez

Rwemalika

Papadakis

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the limitations in the study reduce the capacity of observation, due to i) the tests to evaluate a change were selected according to some structural criterion (coverage, for example), assessing the changes from a different perspective from the original; (ii) the experiment optimized the functions separately, observing improvements only in this isolated context; and (iii) there was no update of these functions in the original software for evaluation and comparison with all of them updated. Harman et al (2014) applied GI in the migration and transplantation of functionalities between software systems in operation. The researchers experimented with using an instant messaging system (Pidgin), and another one of text translation (Babel Fish).…”

Section: Related Workmentioning

confidence: 99%

Challenges on applying genetic improvement in JavaScript using a high-performance computer

Farzat

Barros

Travassos

2018

J Softw Eng Res Dev

View full text Add to dashboard Cite

Genetic Improvement is an area of Search Based Software Engineering that aims to apply evolutionary computing operators to the software source code to improve it according to one or more quality metrics. This article describes challenges related to experimental studies using Genetic Improvement in JavaScript (an interpreted and non-typed language). It describes our experience on performing a study with fifteen projects submitted to genetic improvement with the use of a supercomputer. The construction of specific software infrastructure to support such an experimentation environment reveals peculiarities (parallelization problems, management of threads, etc.) that must be carefully considered to avoid future research threats to validity such as dead-ends, which make it impossible to observe relevant phenomena (code transformation) to the understanding of software improvements and evolution.

show abstract

“…Among them, machine learners and regression algorithms such as Decision Trees, Logistic Regression and Naïve Bayes are widely used [12,36,22]. Recently, also Search-Based approaches have been successfully exploited (e.g., [1,11,25,45,68]). However, according to recent systematic literature reviews [22,66], the choice of a modelling technique seems to have less impact on the classification accuracy of a model than the choice of a metrics set.…”

Section: Software Fault Predictionmentioning

confidence: 99%

“…Moreover our analysis was performed on data belonging to the same software version, thus it is possible that these results might be valid only for the current version. To mitigate this threat we plan to investigate in our future work mutation based metrics both for next-releases [25] and cross-project fault predictions [48,63].…”

Section: Threats To Validitymentioning

confidence: 99%

Mutation-aware fault prediction

Bowes

Hall

Harman

et al. 2016

Proceedings of the 25th International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

David Bowes, Tracy Hall, Mark Harman, Yue Jia, Federica Sarro, and Fan Wu, 'Mutation-aware fault prediction', in Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016. Saarbrucken, Germany, 18-20 July September 2016. Andreas Zeller and Abhik Roychoudhury eds., e-ISBN 978-145034390-9, doi: 10.1145/2931037.2931039. The ACM Digital Library is published by the Association for Computing Machinery. Copyright ?? 2017 ACM, Inc.We introduce mutation-aware fault prediction, which leverages additional guidance from metrics constructed in terms of mutants and the test cases that cover and detect them. We report the results of 12 sets of experiments, applying 4 Different predictive modelling techniques to 3 large real-world systems (both open and closed source). The results show that our proposal can significantly (p ??? 0:05) improve fault prediction performance. Moreover, mutation-based metrics lie in the top 5% most frequently relied upon fault predictors in 10 of the 12 sets of experiments, and provide the majority of the top ten fault predictors in 9 of the 12 sets of experiments

show abstract

Less is More: Temporal Fault Predictive Performance over Multiple Hadoop Releases

Cited by 34 publications

References 17 publications

The importance of accounting for real-world labelling when predicting software vulnerabilities

The importance of accounting for real-world labelling when predicting software vulnerabilities

Challenges on applying genetic improvement in JavaScript using a high-performance computer

Mutation-aware fault prediction

Contact Info

Product

Resources

About