A checklist for integrating student empirical studies with research and teaching goals

Abstract

The use of empirical studies with students in software engineering helps researchers gain insight into new or existing techniques and methods. However, due mainly to concerns of external validity, questions have been raised about the value of these types of studies. The authors of this paper draw on their experiences of conducting a large number of empirical studies in university courses in three countries (Italy, Norway, and the United States) to address this important issue. This paper first identifies the requirements that research and pedagogy place on a valid empirical study with students. This information is then used as the basis for a checklist that provides guidance for researchers and educators when planning and conducting studies in university courses. The goal of this checklist is to help ensure that these studies have as much research and pedagogical value as possible. Finally, an example application of the checklist is provided to illustrate its use

On the Equivalence Between Graphical and Tabular Representations for Security Risk Assessment

“…In regard to external validity, the main issues threatening the generalizability of the results are the use of students instead of practitioners and the use of simple scenarios to apply the methods under evaluation [36]. The use of MSc students in empirical studies is still question of debate.…”

Guidelines for using empirical studies in software engineering education

“…Carver et al discuss costs and benefits for researchers, students, instructors, and industry, and provide a check-list with advice for carrying out empirical studies with student subjects. The same authors have later extended their check-list with requirements for successful empirical studies with students, based on previous literature (Carver et al, 2010). The check-list includes items addressing considerations before a class begins, as soon as it begins, when the study begins, and when the study is completed.…”

Sharing Developers' Mental Models through Source Code Annotations

“…Carver et al [12] recommend to integrate software engineering experiments performed on students with teaching goals. We decided to execute the experiment as a part of the Modeling and Generation of Software Architectures course, which contained Java annotations in its curricula.…”