Conducting realistic experiments in software engineering

Sjøberg, Dag I. K.; Anda, Bente; Arisholm, Erik; Dybå, Tore; Jørgensen, Magne; Karahasanovic, Amela; Koren, Espen Frimann; Vokac, Marek

doi:10.1109/isese.2002.1166921

Cited by 122 publications

(88 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We chose MagicDraw over other modeling tools because it fully supports UML class diagrams and state machines and it is intuitive to use. Using a tool makes the modeling task more realistic than pen-andpaper only [9]. It has the additional advantage that it prevents participants from making syntactic mistakes.…”

Section: A Context and Materialsmentioning

confidence: 99%

“…Alternatively, we could have recruited these students with some financial incentives. According to Sjøberg et al [9], our experiment would have been easier to organize with this alternative. However, some students could have refused to participate in our experiment, reducing the number of data points and introducing a bias towards motivated students.…”

Section: A Context and Materialsmentioning

confidence: 99%

See 1 more Smart Citation

Impact of footprinting on model quality: An experimental evaluation

Jeanneret

Glinz

Baudry

et al. 2012

2012 Second IEEE International Workshop on Model-Driven Requirements Engineering (MoDRE)

View full text Add to dashboard Cite

When modeling requirements, software analysts have to choose the relevant modeling constructs among all those available. If they do not choose the right set, their model may lack some important information or their model may contain many superfluous details. In previous work, we proposed to capture the purpose of a model with a set of model operations such as queries or model transformations. Then, modelers can analyze the footprints of these operations, that is, the set of model elements touched during their execution.In this paper, we report on two controlled experiments performed with students to evaluate whether footprinting can help them in creating better models. While our studies did not demonstrate statistically significant benefits of footprinting, they reveal the importance of training and tool support for the analysis of footprints. In this paper, we report on two controlled experiments performed with students to evaluate whether footprinting can help them in creating better models. While our studies did not demonstrate statistically significant benefits of footprinting, they reveal the importance of training and tool support for the analysis of footprints.

show abstract

Section: A Context and Materialsmentioning

confidence: 99%

Section: A Context and Materialsmentioning

confidence: 99%

Impact of footprinting on model quality: An experimental evaluation

Jeanneret

Glinz

Baudry

et al. 2012

2012 Second IEEE International Workshop on Model-Driven Requirements Engineering (MoDRE)

View full text Add to dashboard Cite

show abstract

“…3.5). All these methodological steps were based on practical guidelines on empirical studies [42,45].…”

Section: Study Methodologymentioning

confidence: 99%

Effects of stability on model composition effort: an exploratory study

2013

View full text Add to dashboard Cite

Model composition plays a central role in many software engineering activities, e.g., evolving design models to add new features. To support these activities, developers usually rely on model composition heuristics. The problem is that the models to-be-composed usually conflict with each other in several ways and such composition heuristics might be unable to properly deal with all emerging conflicts. Hence, the composed model may bear some syntactic and semantic inconsistencies that should be resolved. As a result, the production of the intended model is an error-prone and effortconsuming task. It is often the case that developers end up examining all parts of the output composed model instead of prioritizing the most critical ones, i.e., those that are likely to be inconsistent with the intended model. Unfortunately, little is known about indicators that help developers (1) to identify which model is more likely to exhibit inconsistencies, and (2) to understand which composed models require more effort to be invested. It is often claimed that software systems remaining stable over time tends to have a lower number of defects and require less effort to be fixed than unstable systems. However, little is known about the effects of software stability in the context of model evolution when supported by composition heuristics. This paper, therefore, presents an exploratory study analyzing stability as an indiCommunicated by Prof. Lionel Briand.K. Farias (B) · A. Garcia · C. Lucena OPUS Research Group, LES, Informatics Department, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, RJ, Brazil e-mail: kleinner@gmail.com; kfarias@inf.puc-rio.br A. Garcia e-mail: afgarcia@inf.puc-rio.br C. Lucena e-mail: lucena@inf.puc-rio.br cator of inconsistency rate and resolution effort on model composition activities. Our findings are derived from 180 compositions performed to evolve design models of three software product lines. Our initial results, supported by statistical tests, also indicate which types of changes led to lower inconsistency rate and lower resolution effort.

show abstract

“…The main arguments center around a lack of empirical evaluations (Tichy et al 1995;Glass et al 2002), poor study execution (Kitchenham et al 2002;Dybå et al 2006) and a lack of realism when performed (Sjøberg et al 2002). This may hamper progress in the field, as practitioners looking to adopt new technologies developed in academia are offered scarce decision support (Ivarsson and Gorschek 2009).…”

Section: Introductionmentioning

confidence: 99%

A method for evaluating rigor and industrial relevance of technology evaluations

2010

View full text Add to dashboard Cite

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.

show abstract

Conducting realistic experiments in software engineering

Cited by 122 publications

References 20 publications

Impact of footprinting on model quality: An experimental evaluation

Impact of footprinting on model quality: An experimental evaluation

Effects of stability on model composition effort: an exploratory study

A method for evaluating rigor and industrial relevance of technology evaluations

Contact Info

Product

Resources

About