Crossover Designs in Software Engineering Experiments: Benefits and Perils

Abstract

In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis.Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats. 4 concerning crossover experiment analysis and design, respectively, according to the following schema. First, we adapt the generic principle to SE. Then, we review some real SE experiments that did not apply the good practice and highlight the dangers of not having adhered to the practice. Section 6 provides practical advice by summarizing the suggested good practices for SE researchers running crossover experiments. Section 7 illustrates an application example reporting a crossover experiment that we have conducted and discusses the differences in the results depending on the proper use of this type of design. Finally, Section 8 outlines the conclusions of our research.0098-5589 (c) 6 As Table 1 shows, repeated measures designs, and particularly crossover designs, are useful for addressing two key problems to which SE experiments are commonly prone: small sample sizes and large between-subject variations: 0098-5589 (c)

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

“…In the context of experiments and quasiexperiments, Vegas et al [46] reviewed 39 papers using crossover designs (which are a form of repeated measures design) and found 58% of the papers did not use an analysis method consistent with the design, which could "compromise the validity of the findings". Papers that used the invalid analysis are valueless scientifically, unless their raw data is available for re-analysis.…”

“…It can only ensure that the data and analysis methods are available for inspection and that the results presented in the paper can be derived from the data and analysis procedures. However, if it were adopted, we hope that design and analysis errors, such as those reported by Vegas et al [46] and Shepperd et al [39] would be more likely to be uncovered, hopefully, prior to publication during the review process or soon after publication as the full details will be available to all interested readers. Furthermore, it would provide a valuable resource for training novice researchers.…”

Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review

“…e lack of information about the existence of such problems in SE is rather surprising. e SE methodological literature has not widely addressed this topic; only some works [23,41,52,81] have scratched the surface. Researchers may not be aware of the existence of statistical errors, much less their prevalence and potential impact.…”

“…When we searched for SE papers related to statistical problems, we only found the following results: Dybå et al 's paper regarding statistical power [23], Miller's paper on meta-analysis [52], and two papers by Kitchenham [41] and Vegas et al [81] that focused on within-subject designs.…”

“…ere are works in SE discussing statistical power [23], heterogeneity in meta-analysis [52], and the relative strengths and weaknesses of cross-over designs [41,81]. is stands in contrast with the other disciplines where it is relatively easy to nd works warning about problems in simple statistical concepts such as the de nition of hypotheses [16,58], interpretation of p-values [62], sample size calculation [2,25] and signi cance levels [58], to cite a few.…”