A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications

Abstract

Researchers from different groups and institutions are collaborating on building groups of experiments by means of replication (i.e., conducting groups of replications). Disparate aggregation techniques are being applied to analyze groups of replications. The application of unsuitable techniques to aggregate replication results may undermine the potential of groups of replications to provide in-depth insights from experiment results. Objectives: Provide an analysis procedure with a set of embedded guidelines to aggregate software engineering (SE) replication results. Method: We compare the characteristics of groups of replications for SE and other mature experimental disciplines such as medicine and pharmacology. In view of their differences, the limitations with regard to the joint data analysis of groups of SE replications and the guidelines provided in mature experimental disciplines to analyze groups of replications, we build an analysis procedure with a set of embedded guidelines specifically tailored to the analysis of groups of SE replications. We apply the proposed analysis procedure to a representative group of SE replications to illustrate its use. Results: All the information contained within the raw data should be leveraged during the aggregation of replication results. The analysis procedure that we propose encourages the use of stratified individual participant data and aggregated data in tandem to analyze groups of SE replications. Conclusion: The aggregation techniques used to analyze groups of replications should be justified in research articles. This will increase the reliability and transparency of joint results. The proposed guidelines should ease this endeavor.

Comparing techniques for aggregating interrelated replications in software engineering

“…Although statistical techniques such as sub-group meta-analysis or linear mixed models are key for assessing the effect of changes made across replications [8] [69]. Suppose, for example, we run one replication with professionals and another with students, any statistically significant difference between the results of both replications identified by the sub-group meta-analysis may potentially have been caused by variations in the age of the participants (as professionals tend to be older than students), their motivation (as professionals may be less motivated than students), or their treatment conformance (as professionals may have had to attend to their daily work activities, which may have prevented them from adhering closely to the treatment application procedure) and so on rather than to the different types of participants across the replications.…”

Proceso para la agregación de familias de experimentos en Ingeniería del Software

“…Únicamente existe un estudio que investiga específicamente la agregación de familias de experimentos (Santos, et al, 2019) en el que proponen un método para analizar familias de experimentos compuesto de 4 fases:…”

“…Además (Santos, et al, 2019), han establecido 6 guías para los dos últimos pasos. Las guías para agregar resultados son:  Guía 1.…”

“…Por su parte, un estudio del arte sobre el análisis de familias de experimentos arroja que no existe apenas investigación acerca de cómo deben analizarse las familias de experimentos en IS (Santos, et al, 2019), tal y como se muestra en el Capítulo 2.…”

“…La Tabla 2.15 realiza una comparativa entre el proceso propuesto por Ciolkowski (Ciolkowski, 2012) y Santos (Santos, et al, 2019), y el que aquí se propone. Por lo tanto, tal y como se ha podido observar:…”