A method for evaluating rigor and industrial relevance of technology evaluations

Abstract

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.

Goal-oriented requirements engineering: an extended systematic mapping study

“…As our survey is designed as an SMS and not an SLR, we do not perform a deeper evaluation of paper quality, for example using criteria provided by Ivarsson and Gorschek [22]. In the trade-off between paper volume and survey depth, in choosing to conduct a SMS, we focus on volume, covering many papers in a shallow way.…”

Open innovation in software engineering: a systematic mapping study

“…We used a comprehensive set of guidelines that cover rigor and relevance of studies. We slightly tailored the criteria from Ivarsson et al [31] to fit into the scope of this study, see Section 3.4.…”

“…We used the assessment checklist for rigor and relevance of the research, proposed by Ivarsson et al [31]. Two researchers reviewed the ratings and data extraction to ensure objectivity.…”

“…Therefore, we identified a need to systematically review OI research in SE with a specific focus on assessing the strength of the empirical evidence in the identified studies [31], highlighting the current themes and outlining implications for research and practice. For instance, a study might have high relevance (e.g.…”

Reporting experiments to satisfy professionals’ information needs

“…The motivation of the SE research community is to provide decision makers with relevant state-of-the-art technologies they can choose from to reduce risks that might jeopardize the success of their software projects. For years, it has been argued that decision makers in industry hesitate to introduce new technologies if evidence about their benefits and risks is not available, not convincing (Pfleeger and Menezes 2000), not communicated in the right language (Glass 2006), or if they lack relevance and rigor (Ivarsson and Gorschek 2011). A major ambition of empirical SE research is to provide the decision makers in software development with the type of evidence about technologies they need to support informed decision-making when introducing new technologies.…”