Investigations about replication of empirical studies in software engineering: A systematic mapping study

Magalhães, Cleyton V. C.; Silva, Fábio Q. B. da; Santos, Ronnie E. S.; Suassuna, Marcos

doi:10.1016/j.infsof.2015.02.001

Cited by 25 publications

(28 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Replication of empirical studies in Software Engineering (SE) is an essential activity for achieving greater validity and reliability in research results [2], [13]. Most definitions consider replications to be repetitions of research procedures already performed in a so-called original or baseline studies [6]. As many other processes in SE, replications can be classified according to several criteria.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

et al. 2020

View full text Add to dashboard Cite

In any discipline, replications of empirical studies are necessary to consolidate the acquired knowledge. In Software Engineering, replications have been reported since the 1990s, although their number is still small. The difficulty in publishing, the lack of guidelines, and the unavailability of replication packages are pointed out by the community as some of the main causes. Objective: Understanding the current state of replications in Software Engineering studies by evaluating current trends and evolution during the last 6 years. Method: A Systematic Mapping Study including articles published in the 2013-2018 period that report at least one replication of an empirical study in Software Engineering. Results: 137 studies were selected and analysed, identifying: i) forums; ii) authors, co-authorships and institutions; iii) most cited studies; iv) research topics addressed; v) empirical methods used; vi) temporal distribution of publications; and vii) distribution of studies according to research topics and empirical methods. Conclusions: According to our results, the most relevant forums are the Empirical Software Engineering and Information and Software Technology journals, and the Empirical Software Engineering and Measurement conference. We observed that, as in previous reviews by other researchers, most of the studies were carried out by European institutions, especially Italian, Spanish, and German researchers and institutions. The studies attracting more citations were published mainly in journals and in the International Conference on Software Engineering. Testing, requirements, and software construction were the most frequent topics of replication studies, whereas the usual empirical method was the controlled experiment. On the other hand, we identified research gaps in areas such as software engineering process, software configuration management, and software engineering economics. When analysed together with previous reviews, there is a clear increasing trend in the number of published replications in the 2013-2018 period. INDEX TERMS Empirical software engineering, replications, systematic mapping study.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Our goal is to highlight the potential research opportunities regarding both research topics and empirical methods used by experimenters. Since there were reviews on replications covering previous years [4], [6], [7], only studies published since 2013 were considered.…”

Section: Introductionmentioning

confidence: 99%

Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

et al. 2020

View full text Add to dashboard Cite

show abstract

“…As has been noted by others such as de Magalhães et al [62], one of the difficulties we encountered was that there is no consistent interpretation of the notion of replication. For inclusion in our review we required four things (as discussed in Section 2 on page 4).…”

Section: Descriptionmentioning

confidence: 78%

“…The idea is that an empirical result that can be replicated is more trustworthy. This has gained considerable traction and a series of systematic reviews [99,62] have located 135 articles, just up until 2012, describing replication experiments within the domain of empirical software engineering.…”

Section: Meta-analysismentioning

confidence: 99%

The role and value of replication in empirical software engineering results

Shepperd

Ajienka

Counsell

2018

Information and Software Technology

View full text Add to dashboard Cite

Context: Concerns have been raised from many quarters regarding the reliability of empirical research findings and this includes software engineering. Replication has been proposed as an important means of increasing confidence. Objective: We aim to better understand the value of replication studies, the level of confirmation between replication and original studies, what confirmation means in a statistical sense and what factors modify this relationship. Method: We perform a systematic review to identify relevant replication experimental studies in the areas of (i) software project effort prediction and (ii) pair programming. Where sufficient details are provided we compute prediction intervals. Results: Our review locates 28 unique articles that describe replications of 35 original studies that address 75 research questions. Of these 10 are external, 15 internal and 3 internal-same-article replications. The odds ratio of internal to external (conducted by independent researchers) replications of obtaining a 'confirmatory' result is 8.64. We also found incomplete reporting hampered our ability to extract estimates of effect sizes. Where we are able to compute replication prediction intervals these were surprisingly large. Conclusion: We show that there is substantial evidence to suggest that current approaches to empirical replications are highly problematic. There is a consensus that replications are important, but there is a need for better reporting of both original and replicated studies. Given the low power and incomplete reporting of many original studies, it can be unclear the extent to which a replication is confirmatory and to what extent it yields additional knowledge to the software engineering community. We recommend attention is switched from replication research to meta-analysis.

show abstract

“…The role of replication in SE has been largely argued (Bezerra et al 2015;Da Silva et al 2014;de Magalhâes et al 2015;Gómez et al 2014;Kitchenham 2008). Phrases such as "...replication is the repetition of an experiment to double-check its results..." (Juristo and Vegas 2009) or "...a replication is a study that is run,..., whose goal is to either verify or broaden the applicability of the results of the initial study..." (Shull et al 2004) are common in the literature.…”

Section: Replication In Sementioning

confidence: 99%

Increasing validity through replication: an illustrative TDD case

et al. 2020

View full text Add to dashboard Cite

Software engineering (SE) experiments suffer from threats to validity that may impact their results. Replication allows researchers building on top of previous experiments' weaknesses and increasing the reliability of the findings. Illustrating the benefits of replication to increase the reliability of the findings and uncover moderator variables. We replicate an experiment on test-driven development (TDD) and address some of its threats to validity and those of a previous replication. We compare the replications' results and hypothesize on plausible moderators impacting results. Differences across TDD replications' results might be due to the operationalization of the response variables, the allocation of subjects to treatments, the allowance to work outside the laboratory, the provision of stubs, or the task. Replications allow examining the robustness of the findings, hypothesizing on plausible moderators influencing results, and strengthening the evidence obtained.

show abstract

Investigations about replication of empirical studies in software engineering: A systematic mapping study

Cited by 25 publications

References 11 publications

Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

The role and value of replication in empirical software engineering results

Increasing validity through replication: an illustrative TDD case

Contact Info

Product

Resources

About