Evaluating requirements modeling methods based on user perceptions: A family of experiments

Abstract

Numerous methods and techniques have been proposed for requirements modeling, although very few have had widespread use in practice. One drawback of requirements methods is that they lack proper empirical evaluations. This means that there is a need for evaluation methods that consider both the theoretical and practical aspects of this type of methods and techniques. In this paper, we present a method for evaluating the quality of requirements modeling methods based on user perceptions. The method consists of a theoretical model that explains the relevant dimensions of quality for requirements methods, along with a practical instrument with which to measure these quality dimensions Basically, it allows us to predict the acceptance of a particular requirements method in practice, based on the effort of applying the method, the quality of the requirements artifacts produced, and the user perceptions with regard to the quality of the method. The paper also presents an empirical test of the method that has been proposed for evaluating a Rational Unified Process (RUP) extension for requirements modelling. That test was carried out through a family of experiments conducted with students and practitioners and provides evidence of the usefulness of the evaluation method proposed.

Empirical validation of a usability inspection method for model-driven Web development

“…Dealing with experimental replications has been addressed by the concept of the family of experiments. Although many empirical studies of this type have been applied in the Software Engineering field (e.g., CruzLemus et al [2011]; Abrahão et al [2011]), few families of experiment have been reported in the Web Engineering field (e.g., Abrahão and Poels [2009]). Another issue also appears which is specific to the Web Engineering field: the majority of empirical studies cannot be considered to be methodologically rigorous.…”

Understandability of Goal Concepts by Requirements Engineering Experts

“…How easy is it to construct a model in a language? This concept of understanding is used by, for example, Carvallo & Franch [6] and by Matulevičius & Heymans [13], who measured the number of mistakes made in constructing i* models, and by Abrahao et al, who measured the time needed to build a model [1]. Our concept of understandability is close to the first of these, and we define the understandability of a language construct as the percentage of users that can use a concept correctly.…”

“…To test the generalizability of these findings, we have replicated the experiment with participants of the REFSQ '14 conference that can be considered experts in GORE languages 1 . We additionally asked the subjects for the perceived understandability of ARMOR concepts in an exit survey.…”

A domain-specific visual language for modeling metacognition in intelligent systems

“…Participants performed two metacognitive diagrams and the variables used to measuring the user perception with regard to the quality of the notation are based on (Abrahão, Insfran, Carsí, & Genero, 2011;Wohlin et al, 2000): (i) Perceived ease of use. This variable represents a perceptual judgment of the effort required to use M++.…”