Can we evaluate the quality of software engineering experiments?

Abstract

Context:The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

“…With respect to RQ1, these results suggest that using two reviewers is sufficient providing that there is a period of discussion among the reviewers. These results were similar to those found using the unweighted Kappa [26].…”

“…Initial results for Studies 1 and 2 have already been presented in [26]. However, for this paper the analyses have been changed to use the more appropriate weighted Kappa and ICC reliability metrics.…”

“…In this paper, we report the results of using the Weighted Kappa statistic [9] and the Intra-Class Correlation (ICC) [41]. Note that in our previous paper [26] we only used the simple unweighted Kappa, which is not really appropriate for ordinal scale metrics, and we did not use the ICC metric at all.…”

“…An earlier version of this article was presented at the ESEM 2010 conference [26]. The ESEM paper was based on Studies 1 and 2 alone.…”

Quality indicators for business process models from a gateway complexity perspective

“…Some researchers realized the need to analyze the quality of experiments in an objective way. In [27], authors described an attempt to develop a procedure for evaluating the quality of experiments by means of a quality checklist. This checklist is classified into three groups:  Questions on aims: the aims of our research are clearly stated for each experiment, specifying hypotheses in each case.…”