Understanding replication of experiments in software engineering: A classification

Abstract

Context: Replication plays an important role in experimental disciplines. There are still many uncertain-ties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experiment-ers, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classi-fication which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a litera-ture review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special ver-ification purposes and afford different degrees of change. Each replication type helps to discover partic-ular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.

An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach

“…Therefore, external replications should be preferable as they are less affected by experimenters' biases [24,38]. More recently, Gómez et al [19] propose a different classification, in which the study presented in this paper occurs to be an operational replication. In particular, following Gómez et al, we consider this to be a changed-experimenter replication, because at least two treatments and one response variable are retained, and the researchers are different with respect to the baseline study.…”

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

“…Replication involves repeating an experiment with different participants or experimental materials to investigate whether previous experimental results are repeatable. However as Gomez et al [14] point out many researchers talk about reproducibility in the context of replication. For example, Carver et al talk about results being reproduced [6], and Runeson et al [37] use the term reproduction to refer to a replication performed by independent researchers.…”

Classification of Meteorological Satellite Ground System Applications

“…Thomas Thüm and Juristo N. [1] [2] give a classification and survey of analysis strategies for software product lines. A software product line is a family of software products that share a common set of features.…”