Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

Abstract

Context: During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. [Type text]However, there is little empirical evidence concerning the effectiveness of these guidelines. Aims: This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Method: Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the "teams" of one person as a control. Results: For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. Conclusions: When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

Protocol adaptations to conduct Systematic Literature Reviews in Software Engineering: a chronological study

“…It can give rise to a new type of research for SLR and SE according to the quality of primary works and the need to establish more tertiary studies that are dedicated to reviewing the quality of secondary studies. An example of this are Kitchenham et al (2012) and Zhang and Ali-Babar (2013), which report on issues, activities, working groups characterization and quality of SLRs and primary works collected.…”

Trends in the Quality of Human-Centric Software Engineering Experiments--A Quasi-Experiment

“…The comparison was based on a quality questionnaire described in detail in a previous paper [18]. Seventy papers were selected in such a way that they provided as even a spread of papers per year as possible.…”

“…In a recent paper, we summarised research related to quality criteria used to evaluate experiments [18], pointing out that quality criteria in medical studies were based on three issues:…”

“…• We excluded from the set of candidate papers those papers that we had used to test our quality questionnaire [18].…”

“…With seven researchers available to conduct the study, and a decision that we wanted to obtain quality evaluations from three researchers for each paper (based on the recommendation of [18]), we decided to limit ourselves to 35 recent papers and 35 older papers. This meant that each researcher would have to assess 30 papers.…”