Robust Statistical Methods for Empirical Software Engineering

Kitchenham, Barbara; Madeyski, Lech; Budgen, David; Keung, Jacky; Brereton, Pearl; Charters, Stuart; Gibbs, Shirley; Pohthong, Amnart

doi:10.1007/s10664-016-9437-5

Cited by 189 publications

(170 citation statements)

References 58 publications

(67 reference statements)

Supporting

Mentioning

161

Contrasting

Unclassified

Order By: Relevance

“…We decided to develop the reproducer R package [27] and made it available from CRAN -the official repository of R packages. Data sets analyzed in our three research papers [20,22,28] are encapsulated in the reproducer R package, while most of the figures and tables (particularly those which depend on data), as well as computational results, are built on the fly from data sets stored in the reproducer package and automatically exported into the manuscript rather than copied. -All references were stored in the pure BibTeX format.…”

Section: A Reproducible Research Environmentmentioning

confidence: 99%

See 1 more Smart Citation

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Madeyski

Kitchenham

2017

IFS

Self Cite

View full text Add to dashboard Cite

Abstract.Researchers have identified problems with the validity of software engineering research findings. In particular, it is often impossible to reproduce data analyses, due to lack of raw data, or sufficient summary statistics, or undefined analysis procedures. The aim of this paper is to raise awareness of the problems caused by unreproducible research in software engineering and to discuss the concept of reproducible research (RR) as a mechanism to address these problems. RR is the idea that the outcome of research is both a paper and its computational environment. We report some recent studies that have cast doubts on the reliability of research outcomes in software engineering. Then we discuss the use of RR as a means of addressing these problems. We discuss the use of RR in software engineering research and present the methodology we have used to adopt RR principles. We report a small working example of how to create reproducible research. We summarise advantages of and problems with adopting RR methods. We conclude that RR supports good scientific practice and would help to address some of the problems found in empirical software engineering research.

show abstract

Section: A Reproducible Research Environmentmentioning

confidence: 99%

“…Researchers with more experience of RR concepts can view the reproducer R package [27] including data sets analyzed in our recently published research papers [20,22,28].…”

Section: An Example Of the Rr Processmentioning

confidence: 99%

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Madeyski

Kitchenham

2017

IFS

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, Kitchenham's paper introduced robust statistical methods [42], while Arcuri and Briand's paper discussed statistical tests for the assessment of randomized algorithms [4]. ese works do not assess the weaknesses in current research.…”

Section: Statistical Errors In Sementioning

confidence: 99%

Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review

Reyes

Dieste

Fonseca

et al. 2018

EasyChair Preprints

View full text Add to dashboard Cite

Background: Statistical concepts and techniques are o en applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in so ware engineering (SE). Aim: Assess the existence of statistical errors in SE experiments. Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications. Results: e same errors identi ed in others disciplines were found in ICSE experiments, exhibiting rather large prevalences, over 30% of the reviewed papers in several types of errors such as: a) Missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical tests assumptions, and d) uncorrected multiple testing. When experiments restrict to the validation section of a larger research paper, the prevalence of errors increases.e origin of the errors can be traced back to: a) Researchers' inadequate statistical training, and, b) abundance of exploratory research. Conclusions: is paper provides preliminary evidence that SE research su ers the same statistical problems than other experimental disciplines. However, SE community does not seem to be aware of the existence of shortcomings in their experiments, whereas other disciplines work hard to avoid them. Further research is necessary to nd the underlying causes and set corrective measures, but at the outset some actions could be e ective: a) Improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications. CCS CONCEPTS•General and reference →Surveys and overviews;

show abstract

“…It provides an unbiased measure that favors model generalization. A robust statistical test called the Welch t-test statistic as recommended by Kitchenham et al [19] is also used to determine the statistical pairwise differences among the weighted (Triangular, Epanechnikov and Gaussian) and unweighted (Rectangular) windows [10] and the growing portfolio. We considered the Kruskal-Wallis H-test statistic [4] to find the existence of statistical difference among the four weighting functions applied for both the moving window and the growing portfolio.…”

Section: B Experimental Setup and Evaluation Measuresmentioning

confidence: 99%

A Stratification and Sampling Model for Bellwether Moving Window

Mensah

Keung

Bosu³

et al. 2017

International Conferences on Software Engineering and Knowledge Engineering

Self Cite

View full text Add to dashboard Cite

Abstract-An effective method for finding the relevant number (window size) and the elapsed time (window age) of recently completed projects has proven elusive in software effort estimation. Although these two parameters significantly affect the prediction accuracy, there is no effective method to stratify and sample chronological projects to improve prediction performance of software effort estimation models. Exemplary projects (Bellwether) representing the training set have been empirically validated to improve the prediction accuracy in the domain of software defect prediction. However, the concept of Bellwether and its effect have not been empirically proven in software effort estimation as a method of selecting exemplary/relevant projects with defined window size and age. In view of this, we introduce a novel method for selecting relevant and recently completed projects referred to as Bellwether moving window for improving the software effort prediction accuracy. We first sort and cluster a pool of N projects and apply statistical stratification based on Markov chain modeling to select the Bellwether moving window. We evaluate the proposed approach using the baseline Automatically Transformed Linear Model on the ISBSG dataset. Results show that (1) Bellwether effect exist in software effort estimation dataset, (2) the Bellwether moving window with a window size of 82 to 84 projects and window age of 1.5 to 2 years resulted in an improved prediction accuracy than the traditional approach.

show abstract

Robust Statistical Methods for Empirical Software Engineering

Cited by 189 publications

References 58 publications

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review

A Stratification and Sampling Model for Bellwether Moving Window

Contact Info

Product

Resources

About